Open-source alternatives to TraceAI

Promptflow is a development framework and orchestrator for building applications powered by large language models. It functions as a suite of tools for designing, orchestrating, and deploying AI workflows by linking prompts, custom Python code, and language models into executable sequences. The project is distinguished by a visual AI workflow designer that allows for the creation of directed acyclic graphs of logic nodes. It provides a dedicated prompt engineering environment for versioning and comparing templates, alongside stateful execution tracing to record function calls and variable val

Evidently is an AI observability platform and evaluation framework designed to quantify the performance of machine learning models and large language models. It functions as a monitoring tool for detecting data drift and quality degradation in tabular datasets, while providing a specialized analyzer for the faithfulness and correctness of retrieval augmented generation systems. The project distinguishes itself through an evaluation framework that utilizes judge models and custom rubrics to score language model outputs. It includes tools for iterative prompt optimization and the generation of

Fiddler Auditor is a tool to evaluate language models.

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.

Production-grade AI evaluation, prompt management & observability SDK. Automated evaluations with sub-100ms guardrails. No human-in-the-loop required. Python TypeScript.

Giskard is an evaluation framework, testing library, and quality monitoring system for large language models and AI agents. It serves as a toolkit for quantifying model performance and reliability, providing specialized capabilities for validating retrieval-augmented generation pipelines. The project distinguishes itself through an automated red teaming tool and security scanner designed to identify vulnerabilities, prompt injections, and safety risks. It utilizes adversarial probing and synthetic edge case generation to quantify model robustness and detect information disclosure. The platfo

Helicone is an AI gateway and observability platform designed to intercept, manage, and monitor interactions with large language models. By acting as a reverse-proxy, it provides a centralized layer for routing requests across multiple AI providers, allowing developers to maintain consistent application logic while gaining deep visibility into model performance, usage, and costs. The platform distinguishes itself through a robust suite of traffic management and prompt engineering tools. It enables policy-driven control, including automatic failover between providers, rate limiting, and edge-b

Regression testing for AI agents. Snapshot behavior,diff tool calls,catch regressions in CI. Works with LangGraph, CrewAI, OpenAI, Anthropic.

Runtime supervisor for multi-agent Shared Sessions with Execution Rings, Joint Liability, and Saga Orchestration

Reliability Engineering for AI Agent Systems

EXPLAIN ANALYZE for RAG retrieval — inspect, debug, benchmark, and tune your retrieval layer

Manifest is a language model provider unification system that standardizes access to multiple AI backends through a single interface. It functions as a centralized management layer for integrating various cloud-based and local model providers to simplify how applications request completions. The system provides intelligent model routing and high availability infrastructure by directing queries based on complexity and automatically triggering model fallbacks when a primary provider fails. It distinguishes itself through multi-tenant AI management, organizing agents into isolated groups with de

Track AI API quotas across Synthetic, Z.ai, Anthropic (Claude Code), Codex, GitHub Copilot & Antigravity in real time. Lightweight background daemon (<50MB RAM), SQLite storage, Material Design 3 dashboard. Zero telemetry.

AISecOps (AI Security Operations) framework for deterministic verification of AI systems. QWED verifies LLM outputs using math, logic, and symbolic execution — creating an auditable trust boundary for agentic AI systems. Not generation. Verification.

Automated detection of knowledge gaps and blind spots in RAG vector stores.

Voight observability for the Vercel AI SDK. An OpenTelemetry SpanExporter that ingests the experimental_telemetry spans produced by streamText / generateText / streamObject / generateObject — prompts, tokens, tool calls, cache reads, latency, errors — surfaced live in the Voight dashboard.

"Batteries included" logging solution for your Azure OpenAI instance.

Deepchecks is a machine learning model validation framework and MLOps testing library. It serves as an AI data quality suite and performance evaluator designed to verify the integrity and performance of models and datasets from research through production. The project functions as a model monitoring tool for tracking data drift and performance degradation in production environments. It allows for the creation of custom validation suites and utilizes a pluggable check architecture to automate quality checks within continuous integration pipelines. The framework covers a broad range of capabil

Transparent proxy for any MCP server. Intercepts all JSON-RPC messages, measures latency, stores traces in a configurable database, and enforces per-server allow/deny policies — without touching the original server.