How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
Ragas is an evaluation framework and performance benchmark designed to quantify the quality of retrieval augmented generation pipelines. It functions as an application optimizer to identify bottlenecks in language model workflows using automated metrics and model-based scoring. The framework includes a system for generating synthetic datasets that mimic production scenarios and edge cases to create realistic test cases. It enables reference-free assessment, allowing the evaluation of response quality by analyzing grounding in the provided context without requiring gold-standard labels. The s
Evaluation and Tracking for LLM Experiments and AI Agents
Deepeval is a framework for testing and evaluating large language model applications. It provides a suite of tools for executing automated regression tests, validating model output quality against defined standards, and tracing the execution of complex agent workflows. By integrating these capabilities into development pipelines, the platform ensures consistent performance and reliability throughout the software lifecycle. The platform distinguishes itself through its focus on programmatic validation and observability. It utilizes secondary language models to score output quality and employs
ParlAI is a conversational AI research framework designed for training, evaluating, and sharing dialogue models using a unified interface for datasets and agents. It functions as a PyTorch-based training platform and a dialogue data collection system, providing a centralized model zoo for the distribution of versioned pretrained agents. The project distinguishes itself through a knowledge-grounded retrieval system that combines dense and sparse indexing to ground responses in external information. It also provides a comprehensive infrastructure for gathering human-AI interaction data via inte
A framework for the evaluation of autoregressive code generation language models.
The main features of bigcode-project/bigcode-evaluation-harness are: Model Evaluation and Benchmarking, Evaluation Frameworks.
Projects with overlapping indexed features include: confident-ai/deepeval — Deepeval is a framework for testing and evaluating large language model applications. It provides a suite of tools for… truera/trulens — Evaluation and Tracking for LLM Experiments and AI Agents. explodinggradients/ragas — Ragas is an evaluation framework and performance benchmark designed to quantify the quality of retrieval augmented… facebookresearch/parlai — ParlAI is a conversational AI research framework designed for training, evaluating, and sharing dialogue models using… infrasys-ai/aiinfra. open-mmlab/mmsegmentation — MMSegmentation is an open-source semantic segmentation toolbox built on PyTorch that provides a modular, configurable…