7 repositorios
Awesome GitHub RepositoriesLLM Execution Tracing
Specialized tracing for capturing model-specific data like prompts, completions, and token usage.
Distinguishing note: Focuses on LLM-specific telemetry.
Explore 7 awesome GitHub repositories matching artificial intelligence & ml · LLM Execution Tracing. Refine with filters or upvote what's useful.
Awesome LLM Execution Tracing GitHub Repositories
Encuentra los mejores repositorios con IA.Buscaremos los repositorios que mejor coincidan usando IA.
- openai/openai-agents-python
openai/openai-agents-python
27,191Ver en GitHubThis project is a Python framework for building autonomous, event-driven agent systems. It provides a unified runtime for orchestrating multi-agent workflows, managing persistent conversation state, and executing code within secure, isolated sandbox environments. The framework is designed to handle complex task delegation, allowing agents to invoke other agents as tools while maintaining context across multi-turn interactions. The framework distinguishes itself through its deep integration with the Model Context Protocol, enabling agents to connect to external data sources and remote services
Records timing and metadata for individual operations like LLM calls and tool executions.
Pythonagentsaiframework - mlflow/mlflow
mlflow/mlflow
26,554Ver en GitHubCaptures detailed model call data including prompts, completions, and token counts.
Pythonagentopsagentsai - boundaryml/baml
BoundaryML/baml
7,636Ver en GitHubBAML is a prompt engineering framework and LLM client generator that defines AI prompts as type-safe functions. It serves as a structured data extraction tool and workflow orchestrator, transforming unstructured model responses into strongly typed objects using a custom schema language and alignment algorithms. The project distinguishes itself by using a compiler to generate language-specific boilerplate code for API communication and output parsing. It features a dedicated environment for designing complex prompt templates with conditional logic and reusable snippets, and employs genetic alg
Monitors token usage, inspects raw API requests, and traces AI function executions for debugging and optimization.
Rustbamlboundarymlguardrails - evidentlyai/evidently
evidentlyai/evidently
7,137Ver en GitHubEvidently is an AI observability platform and evaluation framework designed to quantify the performance of machine learning models and large language models. It functions as a monitoring tool for detecting data drift and quality degradation in tabular datasets, while providing a specialized analyzer for the faithfulness and correctness of retrieval augmented generation systems. The project distinguishes itself through an evaluation framework that utilizes judge models and custom rubrics to score language model outputs. It includes tools for iterative prompt optimization and the generation of
Captures model-specific telemetry, including prompts and completions, to reconstruct execution paths and debug AI workflows.
Jupyter Notebookdata-driftdata-qualitydata-science - aimhubio/aim
aimhubio/aim
6,159Ver en GitHubAim is an open-source platform for logging, visualizing, and comparing machine learning training runs and LLM traces. It provides a remote tracking server and a comparison UI, functioning as an ML experiment tracker, AI workflow logger, and LLM trace recorder that captures prompts, generations, and tool calls from AI applications. The platform distinguishes itself through a run-based data model with local SQLite storage, real-time metric streaming, and a plugin-based explorer system that supports specialized visual analysis of metrics, images, audio, and text. It offers a Python SDK with cont
Captures prompts, generations, and tool calls from LLM workflows for debugging and comparison.
Python - latitude-dev/latitude-llm
latitude-dev/latitude-llm
4,145Ver en GitHubEste proyecto es un stack de monitoreo de IA autohospedado que funciona como una plataforma de observabilidad de LLM, framework de evaluación de IA y analizador de trazas de OpenTelemetry. Está diseñado para capturar y analizar trazas, sesiones y telemetría de LLM para monitorear el rendimiento de agentes de IA. La plataforma se distingue como un servidor de Model Context Protocol, exponiendo funciones de espacio de trabajo como herramientas para agentes de codificación de IA. Permite la conversión de trazas de producción fallidas en conjuntos de datos de prueba para pruebas de regresión y utiliza clustering de sesiones basado en semántica para descubrir patrones de comportamiento de usuario emergentes. El sistema cubre áreas de capacidad amplias, incluyendo recolección de telemetría para rutas de ejecución de agentes, puntuación de evaluación automatizada para tráfico en vivo y búsqueda semántica para aislar patrones de interacción. También proporciona alertas para regresiones de señal, análisis de comportamiento para fallos de herramientas y redacción de PII para datos de telemetría. El software puede desplegarse en infraestructura privada como una instalación de un solo host o como un cluster escalable utilizando Docker Compose, Kubernetes o Helm charts.
Implements specialized tracing for capturing LLM-specific data including prompts, completions, and token usage.
TypeScript - agenta-ai/agenta
Agenta-AI/agenta
3,860Ver en GitHubAgenta is a Prompt Ops lifecycle manager and prompt management platform that decouples prompt engineering from application code. It serves as a centralized system for developing, versioning, and deploying prompt templates and model configurations across different environments. The platform functions as an AI agent orchestrator with a visual interface for building agent workflows and connecting models to external tools. It further acts as an evaluation framework and observability tool, utilizing OpenTelemetry to capture execution traces, monitor latency, and track token costs. The system cove
Captures detailed LLM execution telemetry including prompts, completions, and token usage.
TypeScriptagentsevaluationllm-as-a-judge
