7 Repos
Awesome GitHub RepositoriesLLM Execution Tracing
Specialized tracing for capturing model-specific data like prompts, completions, and token usage.
Distinguishing note: Focuses on LLM-specific telemetry.
Explore 7 awesome GitHub repositories matching artificial intelligence & ml · LLM Execution Tracing. Refine with filters or upvote what's useful.
Awesome LLM Execution Tracing GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- openai/openai-agents-python
openai/openai-agents-python
27,191Auf GitHub ansehenThis project is a Python framework for building autonomous, event-driven agent systems. It provides a unified runtime for orchestrating multi-agent workflows, managing persistent conversation state, and executing code within secure, isolated sandbox environments. The framework is designed to handle complex task delegation, allowing agents to invoke other agents as tools while maintaining context across multi-turn interactions. The framework distinguishes itself through its deep integration with the Model Context Protocol, enabling agents to connect to external data sources and remote services
Records timing and metadata for individual operations like LLM calls and tool executions.
Pythonagentsaiframework - mlflow/mlflow
mlflow/mlflow
26,554Auf GitHub ansehenCaptures detailed model call data including prompts, completions, and token counts.
Pythonagentopsagentsai - boundaryml/baml
BoundaryML/baml
7,636Auf GitHub ansehenBAML is a prompt engineering framework and LLM client generator that defines AI prompts as type-safe functions. It serves as a structured data extraction tool and workflow orchestrator, transforming unstructured model responses into strongly typed objects using a custom schema language and alignment algorithms. The project distinguishes itself by using a compiler to generate language-specific boilerplate code for API communication and output parsing. It features a dedicated environment for designing complex prompt templates with conditional logic and reusable snippets, and employs genetic alg
Monitors token usage, inspects raw API requests, and traces AI function executions for debugging and optimization.
Rustbamlboundarymlguardrails - evidentlyai/evidently
evidentlyai/evidently
7,137Auf GitHub ansehenEvidently is an AI observability platform and evaluation framework designed to quantify the performance of machine learning models and large language models. It functions as a monitoring tool for detecting data drift and quality degradation in tabular datasets, while providing a specialized analyzer for the faithfulness and correctness of retrieval augmented generation systems. The project distinguishes itself through an evaluation framework that utilizes judge models and custom rubrics to score language model outputs. It includes tools for iterative prompt optimization and the generation of
Captures model-specific telemetry, including prompts and completions, to reconstruct execution paths and debug AI workflows.
Jupyter Notebookdata-driftdata-qualitydata-science - aimhubio/aim
aimhubio/aim
6,159Auf GitHub ansehenAim is an open-source platform for logging, visualizing, and comparing machine learning training runs and LLM traces. It provides a remote tracking server and a comparison UI, functioning as an ML experiment tracker, AI workflow logger, and LLM trace recorder that captures prompts, generations, and tool calls from AI applications. The platform distinguishes itself through a run-based data model with local SQLite storage, real-time metric streaming, and a plugin-based explorer system that supports specialized visual analysis of metrics, images, audio, and text. It offers a Python SDK with cont
Captures prompts, generations, and tool calls from LLM workflows for debugging and comparison.
Python - latitude-dev/latitude-llm
latitude-dev/latitude-llm
4,145Auf GitHub ansehenDieses Projekt ist ein selbstgehosteter KI-Monitoring-Stack, der als LLM-Observability-Plattform, KI-Evaluierungs-Framework und OpenTelemetry-Trace-Analyzer fungiert. Es ist darauf ausgelegt, LLM-Traces, Sitzungen und Telemetrie zu erfassen und zu analysieren, um die Leistung von KI-Agenten zu überwachen. Die Plattform zeichnet sich als Model-Context-Protocol-Server aus, der Workspace-Funktionen als Tools für KI-Coding-Agenten bereitstellt. Sie ermöglicht die Umwandlung fehlerhafter Produktions-Traces in Testdatensätze für Regressionstests und nutzt semantisches Sitzungs-Clustering, um aufkommende Benutzerverhaltensmuster zu entdecken. Das System deckt breite Funktionsbereiche ab, einschließlich Telemetrieerfassung für Agenten-Ausführungspfade, automatisierte Evaluierungs-Scoring für Live-Traffic und semantische Suche zur Isolierung von Interaktionsmustern. Es bietet zudem Alarmierung bei Signal-Regressionen, Verhaltensanalyse bei Tool-Fehlern und PII-Redaktion für Telemetriedaten. Die Software kann auf privater Infrastruktur als Single-Host-Installation oder als skalierbarer Cluster mittels Docker Compose, Kubernetes oder Helm-Charts bereitgestellt werden.
Implements specialized tracing for capturing LLM-specific data including prompts, completions, and token usage.
TypeScript - agenta-ai/agenta
Agenta-AI/agenta
3,860Auf GitHub ansehenAgenta is a Prompt Ops lifecycle manager and prompt management platform that decouples prompt engineering from application code. It serves as a centralized system for developing, versioning, and deploying prompt templates and model configurations across different environments. The platform functions as an AI agent orchestrator with a visual interface for building agent workflows and connecting models to external tools. It further acts as an evaluation framework and observability tool, utilizing OpenTelemetry to capture execution traces, monitor latency, and track token costs. The system cove
Captures detailed LLM execution telemetry including prompts, completions, and token usage.
TypeScriptagentsevaluationllm-as-a-judge
