22 个仓库
Utilities for monitoring, tracing, and analyzing the execution flow and performance metrics of autonomous agent interactions.
Distinguishing note: Focuses specifically on the observability and tracing of agentic conversation flows rather than general-purpose application logging.
Explore 22 awesome GitHub repositories matching artificial intelligence & ml · Agent Observability Tools. Refine with filters or upvote what's useful.
This framework provides a development toolkit for building autonomous agents that utilize language models to solve complex, non-deterministic tasks. Its core design centers on a code-executing architecture where agents generate and run Python code snippets to perform logic, data manipulation, and tool interactions. By moving beyond structured data formats, the system enables agents to manage program flow and object state through iterative reasoning cycles. The project distinguishes itself through its focus on code-based agent implementation and secure execution environments. Developers can ch
Provides an interactive interface to observe the step-by-step reasoning process of an agent.
This project is a Python framework for building autonomous, event-driven agent systems. It provides a unified runtime for orchestrating multi-agent workflows, managing persistent conversation state, and executing code within secure, isolated sandbox environments. The framework is designed to handle complex task delegation, allowing agents to invoke other agents as tools while maintaining context across multi-turn interactions. The framework distinguishes itself through its deep integration with the Model Context Protocol, enabling agents to connect to external data sources and remote services
Hooks into agent execution and tool calls to log activity and record usage metrics.
Deepagents is an LLM agent orchestration platform and stateful application server designed for deploying and managing AI agents built with computational graphs. It provides a containerized runtime environment that handles agent execution, state persistence, and the versioning of AI assistants. The platform distinguishes itself through deep integration with the Model Context Protocol, allowing agents to function as servers that expose tools and capabilities to external clients. It features a sophisticated observability suite for capturing execution traces, performing LLM-based evaluations agai
Captures execution traces and evaluates agent performance through specialized monitoring and metrics.
A2A is a standardized framework designed to enable interoperability, discovery, and orchestration among independent artificial intelligence agents. It provides a common communication protocol that allows heterogeneous agents to exchange data, verify identities, and collaborate across diverse programming languages and computing environments. By establishing a unified messaging standard, the project facilitates the creation of complex, multi-agent workflows where tasks are routed and managed between specialized services. The project distinguishes itself through a capability-based architecture t
Verifies that agent implementations adhere to protocol standards using automated testing tools to ensure consistent interoperability across the ecosystem.
The Model Context Protocol SDK is a framework for building clients and servers that connect AI models to external data, tools, and resources using a standardized communication protocol. It provides the foundational libraries and interfaces necessary to establish reliable, transport-agnostic connections between AI agents and external systems, enabling seamless information retrieval and task automation. The SDK distinguishes itself through a robust capability negotiation handshake that ensures compatibility between connected parties before exchanging messages. It supports a pluggable transport
Ensures protocol version compatibility between clients and servers to prevent runtime errors.
Claude Code is a command-line interface and multi-agent orchestration framework designed for autonomous software engineering. It enables AI agents to perform codebase modifications, debugging, and Git workflow management while coordinating multiple specialized agents to decompose and execute complex engineering tasks in parallel. The system distinguishes itself through a high degree of isolation and safety, utilizing Git worktrees to create independent working directories for concurrent agents and implementing a tiered permission system that combines user rules, project policies, and OS-level
Captures the inputs, outputs, and duration of tool calls to debug agent workflows.
NATS Server is a high-performance, lightweight messaging system designed for cloud-native applications, edge computing, and distributed microservices. It functions as a distributed publish-subscribe broker that routes messages using hierarchical, dot-separated subject strings, enabling decoupled communication between services without requiring centralized broker lookups. The system supports core messaging patterns including asynchronous publish-subscribe, request-reply, and load-balanced queue processing. The platform distinguishes itself through a decentralized architecture that eliminates t
Enforces strict protocol checking to identify and reject malformed requests or invalid subject naming patterns.
SWE-agent is an autonomous software engineering platform designed to automate repository maintenance and issue resolution. By orchestrating language models to navigate codebases, diagnose software bugs, and apply fixes, the framework functions as an autonomous agent capable of executing shell commands, editing source code, and managing pull requests within isolated, containerized environments. The platform distinguishes itself through its focus on end-to-end task autonomy and observability. It features a robust trajectory logging system that records every thought, action, and environment obse
Logs every thought, action, observation, and environment state change during an autonomous task to provide a complete audit trail.
PydanticAI is a Python framework designed for building production-grade autonomous agents. It provides a unified interface for interacting with diverse language models, enabling developers to construct agents that perform complex tasks through structured data validation, tool execution, and multi-turn conversation management. The library centers on type-safe schema enforcement, ensuring that model inputs and outputs remain consistent and reliable throughout the agent's lifecycle. The framework distinguishes itself through a robust architecture that emphasizes modularity and testability. It ut
The framework tracks and visualizes the internal steps and decision-making processes of AI agents to simplify debugging and performance analysis.
This project is a comprehensive framework for building and managing autonomous agent systems. It provides a unified architecture for orchestrating multi-agent societies, where specialized agents collaborate through roleplay to decompose and solve complex tasks. The system integrates language models with external environments, enabling agents to perform real-world actions through a standardized tool-calling abstraction layer. The framework distinguishes itself through its focus on iterative reasoning and data reliability. It employs automated feedback loops to refine agent outputs and self-eva
Structures input context and observations to maintain consistent state for agent decision-making.
Ragas is an evaluation framework designed to measure the performance of retrieval-augmented generation pipelines and autonomous agent workflows. It provides a comprehensive suite of tools for benchmarking system outputs, utilizing language models as automated judges to score performance against defined rubrics and reference data. By standardizing inputs, retrieved contexts, and generated responses into a unified schema, the project enables consistent analysis across complex AI applications. The framework distinguishes itself through its ability to generate synthetic test datasets from existin
Measures the accuracy of agent tool usage by calculating F1 scores against expected reference tool calls.
mcp-use is a development framework designed for building, deploying, and managing servers, clients, and autonomous agents using the Model Context Protocol. It provides a comprehensive toolkit for creating servers that expose custom tools, data resources, and prompts to compatible AI agents. The project distinguishes itself by offering a complete lifecycle for protocol-based applications, including a dedicated hosting platform for production servers and a compliance validator to ensure servers meet marketplace publishing requirements. It also features an observability suite for tracing protoco
Implements validation tools to verify that MCP server handshakes and tool definitions conform to protocol specifications.
Hatchet is an open-source durable workflow engine and task orchestration platform. It provides a framework for building and executing fault-tolerant, multi-step pipelines as directed acyclic graphs (DAGs), with automatic retries, scheduling, and real-time observability. The system is built around durable task checkpointing, which persists execution state after each step so work can resume from the last checkpoint after a worker crash or restart, and it supports event-driven task resumption that pauses a task until a matching external event arrives. The platform distinguishes itself through it
Displays each agent tool call as a task run with status, timing, and input/output details in the dashboard.
mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe
Logs MCP tool calls in agentic tool call records and streaming events with prefixed names.
Emits start and end events for every tool call, reporting its name, arguments, success, and duration.
Helicone is an AI gateway and observability platform designed to intercept, manage, and monitor interactions with large language models. By acting as a reverse-proxy, it provides a centralized layer for routing requests across multiple AI providers, allowing developers to maintain consistent application logic while gaining deep visibility into model performance, usage, and costs. The platform distinguishes itself through a robust suite of traffic management and prompt engineering tools. It enables policy-driven control, including automatic failover between providers, rate limiting, and edge-b
Emits start and end events for every tool call, reporting its name, arguments, success, and duration.
Coze-loop 是一个用于大语言模型智能体的优化平台和编排管理套件。它作为一个全面的环境,用于开发、调试、评估和监控 AI 智能体的性能。 该项目提供了一个专用的提示词工程演练场,用于实时迭代和验证模型响应。它包括一个评估框架,针对数据集运行自动化评估,以生成性能指标并验证输出准确性。 该系统通过实时执行追踪和智能体行为的历史分析来涵盖可观测性。它进一步支持生命周期管理,具有分布式调试、模型参数配置和集群部署自定义功能。
Provides utilities for monitoring, tracing, and analyzing the execution flow and performance of autonomous agent interactions.
Inngest is a durable execution framework and event-driven automation engine designed to orchestrate background workflows. It enables developers to build resilient, stateful processes by memoizing function steps, ensuring that long-running tasks can automatically resume from the last successful operation after failures, timeouts, or infrastructure restarts. The platform distinguishes itself through its event-driven architecture, which uses a schema-validated bus to trigger functions and coordinate complex, multi-step logic. It employs an onion-model middleware approach for cross-cutting concer
Runs individual tool calls as isolated, retriable units that support parallel execution and provide granular observability.
Claude Agent SDK is a Python library from Anthropic for building AI agents that use Claude’s tool‑calling, streaming, and session‑management capabilities. It provides a structured framework for intercepting and logging every tool call an agent makes, managing conversation sessions across multiple turns, and controlling which tools the agent is allowed to invoke through configurable permission rules. The SDK distinguishes itself with middleware‑driven tool interception that lets developers block, modify, or require approval for tool calls before or after execution. A permission policy engine e
Logs all tool call activity via MCP for auditing, debugging, and compliance.
Cerbos is an open-source authorization service that provides a centralized, language-agnostic engine for managing access control. It functions as a policy-as-code platform, allowing teams to define, test, and distribute authorization rules using declarative YAML or JSON configurations. By decoupling access logic from application code, it enables consistent permission enforcement across diverse service stacks. The project distinguishes itself through its ability to translate high-level authorization policies into native database query filters. This capability allows applications to enforce sec
Logs intercepted tool calls and event requests to provide visibility into usage patterns and policy evaluation results.