Awesome GitHub RepositoriesObservability and Tracing

Promptflow is a development framework and orchestrator for building applications powered by large language models. It functions as a suite of tools for designing, orchestrating, and deploying AI workflows by linking prompts, custom Python code, and language models into executable sequences. The project is distinguished by a visual AI workflow designer that allows for the creation of directed acyclic graphs of logic nodes. It provides a dedicated prompt engineering environment for versioning and comparing templates, alongside stateful execution tracing to record function calls and variable val

Eino is an AI agent development kit and LLM application framework designed for building autonomous agents and orchestrating complex language model workflows. It serves as a multi-agent orchestration engine and workflow orchestrator, providing a graph-based execution model to route data between models, tools, and retrievers. The framework distinguishes itself through a robust set of multi-agent coordination patterns, including supervisor-led management, sequential flows, and autonomous reasoning loops like ReAct. It features advanced agent execution controls such as active turn preemption, che

Spring AI is an application framework for Java that provides a portable, fluent API for integrating AI models, tools, and vector stores into applications. It wraps multiple AI providers behind a common interface, allowing developers to switch between chat, embedding, image, and speech models without changing application code. The framework includes a chainable chat client API similar to WebClient or RestClient, supports both synchronous and streaming interactions, and offers structured output conversion that transforms unstructured AI responses into strongly-typed Java objects. The framework

Arize Phoenix is an LLM observability platform and evaluation framework designed to capture execution traces and monitor large language model applications. It serves as a prompt management system for versioning and testing templates, and as a self-hosted AI operations infrastructure for managing telemetry and experiments. The platform differentiates itself through a specialized embedding visualization tool used to detect data drift and optimize vector search. It provides a comprehensive evaluation suite that utilizes judge-based evaluators and ground-truth datasets to score model outputs, and

Fluent Bit est un collecteur de logs et de télémétrie unifié cloud-native conçu comme un pipeline de données efficace en ressources. Il ingère des logs, des métriques et des traces provenant de multiples sources, les traitant en temps réel avant d'acheminer les données vers des backends de stockage externes. Le projet fonctionne comme un processeur de flux en temps réel et un processeur de logs OpenTelemetry, capable de transformer et de filtrer les données en utilisant SQL et une logique conditionnelle. Il agit également comme un agent de traçage distribué capable d'échantillonner les traces pour réduire le volume de données tout en préservant les chemins de requête complets. Le système fournit une livraison de données fiable grâce à une mise en mémoire tampon basée sur le système de fichiers et une logique de réessai avec état pour éviter la perte de données lors des pannes. Son architecture modulaire prend en charge des plugins d'entrée et de sortie enfichables, un routage basé sur les métadonnées et la capacité d'étendre les fonctionnalités via des bibliothèques partagées. Le logiciel peut être déployé en tant que conteneur sur différentes architectures CPU et systèmes d'exploitation.

OpenLLMetry is an OpenTelemetry-based observability framework and instrumentation library for generative AI applications. It provides toolsets for tracing and monitoring large language model workflows, capturing telemetry from model providers, agent frameworks, and vector databases using standardized semantic conventions. The project distinguishes itself by providing a specialized evaluation and experimentation suite that associates user feedback and prompt version hashes with specific execution traces. It includes a system for tracking model reasoning paths and enforcing security guardrails

IntentKit is an open-source platform for deploying and managing a collaborative team of AI agents that can work together to complete complex tasks. It provides a self-hosted agent orchestrator that coordinates multiple agents through a modular pipeline of entrypoints, orchestration, and storage, all running as containerized services using Docker Compose or Swarm for production-grade deployment. The platform distinguishes itself by offering a plugin-based system for extending agent capabilities without modifying the core codebase, along with built-in integrations for connecting agents to socia

NeMo-Guardrails is a toolkit for adding programmable safety constraints and dialogue boundaries to large language model conversational systems. It functions as security middleware that intercepts inputs and outputs to block prompt injections, jailbreaks, and sensitive data leaks, while providing a conversational dialogue manager to define structured interaction flows through configuration files. The framework includes a hallucination filter to screen model outputs for factual accuracy and a specialized modeling language for defining conversational flows and constraints. It provides capabiliti

Open Multi-Agent is a TypeScript framework for multi-agent orchestration that decomposes natural language goals into a runtime-generated directed acyclic graph of tasks. It functions as a task orchestrator and workflow state manager, coordinating multiple AI models to execute parallel and sequential operations. The framework is distinguished by a proposer-judge consensus protocol used to validate agent outputs through a quorum of agreement. It employs provider-agnostic model routing to assign specific models to tasks based on roles or execution phases and utilizes state-based workflow checkpo

This is an open-source Python SDK for building and orchestrating production-grade AI agents. It provides a unified framework for creating conversational agents that can use tools, maintain state, and coordinate across multiple language model providers including OpenAI, Anthropic, Google, Amazon Bedrock, and locally-hosted models. The SDK supports multi-agent orchestration through graphs, teams, and swarms, allowing several specialized agents to collaborate on complex tasks. Agents can be composed as callable tools that other agents invoke, and the framework includes policy handlers that inspe

Coze-loop est une plateforme d'optimisation et une suite de gestion d'orchestration pour les agents de grands modèles de langage (LLM). Elle fonctionne comme un environnement complet pour le développement, le débogage, l'évaluation et la surveillance des performances des agents IA. Le projet fournit un terrain de jeu dédié au prompt engineering pour l'itération et la validation en temps réel des réponses du modèle. Il inclut un framework d'évaluation qui exécute des évaluations automatisées sur des jeux de données pour générer des métriques de performance et vérifier la précision des sorties. Le système couvre l'observabilité via le traçage d'exécution en temps réel et l'analyse historique du comportement des agents. Il prend en charge la gestion du cycle de vie avec des capacités de débogage distribué, de configuration des paramètres du modèle et de personnalisation du déploiement en cluster.

ZenML is an extensible machine learning orchestration framework designed to manage the end-to-end lifecycle of data pipelines and AI agent workflows. It functions as a durable orchestrator that executes machine learning tasks as directed acyclic graphs, ensuring that every step is containerized for consistent performance across local, cloud, and hybrid infrastructure. By decoupling pipeline code from underlying compute and storage backends, the platform allows developers to define infrastructure-agnostic stacks that remain portable across diverse environments. The project distinguishes itself

This project is an educational resource and engineering guide for building, deploying, and optimizing large language model applications and production pipelines. It serves as a blueprint for cloud AI infrastructure, providing a framework for orchestrating inference endpoints, data warehouses, and scalable production environments. The repository provides specific implementation patterns for retrieval augmented generation to ground model responses in external data. It includes a training workflow for crawling, structuring, and processing datasets to facilitate model fine-tuning, alongside an ev

Youtu Agent is an open-source framework for building, running, and evaluating autonomous agents powered by large language models. It provides the core infrastructure for creating agents that follow reasoning loops, use toolkits, and coordinate with other agents to solve complex tasks, all managed through YAML-driven configuration files. The framework distinguishes itself through its support for multi-agent orchestration, where a planner agent decomposes tasks and coordinates specialized worker agents, and through its integration with the Model Context Protocol for connecting to external toolk

Cerbos is an open-source authorization service that provides a centralized, language-agnostic engine for managing access control. It functions as a policy-as-code platform, allowing teams to define, test, and distribute authorization rules using declarative YAML or JSON configurations. By decoupling access logic from application code, it enables consistent permission enforcement across diverse service stacks. The project distinguishes itself through its ability to translate high-level authorization policies into native database query filters. This capability allows applications to enforce sec

Pandera is a data pipeline validation framework and statistical type validation tool. It functions as a library for defining and enforcing schemas on datasets to ensure data quality and consistency, specifically providing validation capabilities for Pandas dataframes. The project includes a schema inference tool that automates setup by analyzing existing dataset samples to generate validation schemas. It also serves as a synthetic data generator, creating artificial datasets based on predefined schemas to verify data-producing functions. The framework covers data engineering quality assuranc

Ce projet est une pile de surveillance IA auto-hébergée qui fonctionne comme une plateforme d'observabilité LLM, un framework d'évaluation IA et un analyseur de traces OpenTelemetry. Il est conçu pour capturer et analyser les traces, sessions et télémétries LLM afin de surveiller les performances des agents IA. La plateforme se distingue en tant que serveur Model Context Protocol, exposant les fonctions de l'espace de travail comme outils pour les agents de codage IA. Elle permet la conversion des traces de production défaillantes en jeux de données de test pour les tests de régression et utilise le clustering de sessions basé sur la sémantique pour découvrir les modèles de comportement émergents des utilisateurs. Le système couvre de larges domaines de capacités, notamment la collecte de télémétrie pour les chemins d'exécution des agents, le scoring d'évaluation automatisé pour le trafic en direct et la recherche sémantique pour isoler les modèles d'interaction. Il fournit également des alertes pour les régressions de signaux, l'analyse comportementale pour les échecs d'outils et la rédaction de PII pour les données de télémétrie. Le logiciel peut être déployé sur une infrastructure privée en tant qu'installation à hôte unique ou en tant que cluster évolutif utilisant Docker Compose, Kubernetes ou des charts Helm.

Uptrace is an OpenTelemetry-based observability platform designed to collect, store, and analyze distributed traces, metrics, and logs. It functions as a centralized logging backend, a distributed tracing system, and a metrics engine to monitor application performance and system health. The platform is distinguished by AI-powered operational capabilities, allowing users to query telemetry data and manage monitoring dashboards using natural language. It specifically includes specialized monitoring for generative AI pipelines, tracking token usage and response quality for LLM interactions and r