Awesome GitHub RepositoriesLLM Execution Tracing

This project is a Python framework for building autonomous, event-driven agent systems. It provides a unified runtime for orchestrating multi-agent workflows, managing persistent conversation state, and executing code within secure, isolated sandbox environments. The framework is designed to handle complex task delegation, allowing agents to invoke other agents as tools while maintaining context across multi-turn interactions. The framework distinguishes itself through its deep integration with the Model Context Protocol, enabling agents to connect to external data sources and remote services

BAML is a prompt engineering framework and LLM client generator that defines AI prompts as type-safe functions. It serves as a structured data extraction tool and workflow orchestrator, transforming unstructured model responses into strongly typed objects using a custom schema language and alignment algorithms. The project distinguishes itself by using a compiler to generate language-specific boilerplate code for API communication and output parsing. It features a dedicated environment for designing complex prompt templates with conditional logic and reusable snippets, and employs genetic alg

Evidently is an AI observability platform and evaluation framework designed to quantify the performance of machine learning models and large language models. It functions as a monitoring tool for detecting data drift and quality degradation in tabular datasets, while providing a specialized analyzer for the faithfulness and correctness of retrieval augmented generation systems. The project distinguishes itself through an evaluation framework that utilizes judge models and custom rubrics to score language model outputs. It includes tools for iterative prompt optimization and the generation of

Aim is an open-source platform for logging, visualizing, and comparing machine learning training runs and LLM traces. It provides a remote tracking server and a comparison UI, functioning as an ML experiment tracker, AI workflow logger, and LLM trace recorder that captures prompts, generations, and tool calls from AI applications. The platform distinguishes itself through a run-based data model with local SQLite storage, real-time metric streaming, and a plugin-based explorer system that supports specialized visual analysis of metrics, images, audio, and text. It offers a Python SDK with cont

这是一个自托管的 AI 监控技术栈，集 LLM 可观测性平台、AI 评估框架和 OpenTelemetry 链路分析器于一体。它旨在捕获并分析 LLM 链路、会话和遥测数据，以监控 AI Agent 的性能。 该平台作为 Model Context Protocol 服务器，将工作区功能暴露为 AI 编码 Agent 的工具。它支持将生产环境中的失败链路转换为回归测试数据集，并利用基于语义的会话聚类来发现新兴的用户行为模式。 系统涵盖了广泛的功能领域，包括 Agent 执行路径的遥测采集、实时流量的自动化评估评分，以及用于隔离交互模式的语义搜索。此外，它还提供信号回归告警、工具故障行为分析以及遥测数据的 PII 脱敏功能。 该软件可通过 Docker Compose、Kubernetes 或 Helm charts 部署在私有基础设施上，支持单机安装或可扩展集群部署。

Agenta is a Prompt Ops lifecycle manager and prompt management platform that decouples prompt engineering from application code. It serves as a centralized system for developing, versioning, and deploying prompt templates and model configurations across different environments. The platform functions as an AI agent orchestrator with a visual interface for building agent workflows and connecting models to external tools. It further acts as an evaluation framework and observability tool, utilizing OpenTelemetry to capture execution traces, monitor latency, and track token costs. The system cove