Evidently is an AI observability platform and evaluation framework designed to quantify the performance of machine learning models and large language models. It functions as a monitoring tool for detecting data drift and quality degradation in tabular datasets, while providing a specialized analyzer for the faithfulness and correctness of retrieval augmented generation systems. The project distinguishes itself through an evaluation framework that utilizes judge models and custom rubrics to score language model outputs. It includes tools for iterative prompt optimization and the generation of
Giskard is an evaluation framework, testing library, and quality monitoring system for large language models and AI agents. It serves as a toolkit for quantifying model performance and reliability, providing specialized capabilities for validating retrieval-augmented generation pipelines. The project distinguishes itself through an automated red teaming tool and security scanner designed to identify vulnerabilities, prompt injections, and safety risks. It utilizes adversarial probing and synthetic edge case generation to quantify model robustness and detect information disclosure. The platfo
Deepchecks is a machine learning model validation framework and MLOps testing library. It serves as an AI data quality suite and performance evaluator designed to verify the integrity and performance of models and datasets from research through production. The project functions as a model monitoring tool for tracking data drift and performance degradation in production environments. It allows for the creation of custom validation suites and utilizes a pluggable check architecture to automate quality checks within continuous integration pipelines. The framework covers a broad range of capabil
Helicone is an AI gateway and observability platform designed to intercept, manage, and monitor interactions with large language models. By acting as a reverse-proxy, it provides a centralized layer for routing requests across multiple AI providers, allowing developers to maintain consistent application logic while gaining deep visibility into model performance, usage, and costs. The platform distinguishes itself through a robust suite of traffic management and prompt engineering tools. It enables policy-driven control, including automatic failover between providers, rate limiting, and edge-b
Track AI API quotas across Synthetic, Z.ai, Anthropic (Claude Code), Codex, GitHub Copilot & Antigravity in real time. Lightweight background daemon (<50MB RAM), SQLite storage, Material Design 3 dashboard. Zero telemetry.
Die Hauptfunktionen von onllm-dev/onwatch sind: Model Evaluation and Benchmarking, Command Line Tools, Observability and Evaluation, Nutzungsanalysen.
Open-Source-Alternativen zu onllm-dev/onwatch sind unter anderem: helicone/helicone — Helicone is an AI gateway and observability platform designed to intercept, manage, and monitor interactions with… traceloop/openllmetry — OpenLLMetry is an OpenTelemetry-based observability framework and instrumentation library for generative AI… giskard-ai/giskard — Giskard is an evaluation framework, testing library, and quality monitoring system for large language models and AI… deepchecks/deepchecks — Deepchecks is a machine learning model validation framework and MLOps testing library. It serves as an AI data quality… evidentlyai/evidently — Evidently is an AI observability platform and evaluation framework designed to quantify the performance of machine… facebookresearch/parlai — ParlAI is a conversational AI research framework designed for training, evaluating, and sharing dialogue models using…