6 Repos
Awesome GitHub RepositoriesEvaluation Frameworks
Tools for testing, benchmarking, and monitoring the performance of RAG systems.
Explore 6 awesome GitHub repositories matching part of an awesome list · Evaluation Frameworks. Refine with filters or upvote what's useful.
Awesome Evaluation Frameworks GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- explodinggradients/ragas
explodinggradients/ragas
14,400Auf GitHub ansehenRagas is an evaluation framework and performance benchmark designed to quantify the quality of retrieval augmented generation pipelines. It functions as an application optimizer to identify bottlenecks in language model workflows using automated metrics and model-based scoring. The framework includes a system for generating synthetic datasets that mimic production scenarios and edge cases to create realistic test cases. It enables reference-free assessment, allowing the evaluation of response quality by analyzing grounding in the provided context without requiring gold-standard labels. The s
Framework for evaluating RAG components like faithfulness and relevance.
Python - confident-ai/deepeval
confident-ai/deepeval
13,733Auf GitHub ansehenDeepeval is a framework for testing and evaluating large language model applications. It provides a suite of tools for executing automated regression tests, validating model output quality against defined standards, and tracing the execution of complex agent workflows. By integrating these capabilities into development pipelines, the platform ensures consistent performance and reliability throughout the software lifecycle. The platform distinguishes itself through its focus on programmatic validation and observability. It utilizes secondary language models to score output quality and employs
Unit testing framework for LLM applications using various NLP metrics.
Pythonevaluation-frameworkevaluation-metricsllm-evaluation - truera/trulens
truera/trulens
3,384Auf GitHub ansehenEvaluation and Tracking for LLM Experiments and AI Agents
Tools for evaluating and explaining LLM-based application performance.
Python - uptrain-ai/uptrain
uptrain-ai/uptrain
2,352Auf GitHub ansehenUpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.
Unified platform for evaluating and improving generative AI applications.
Python - bigcode-project/bigcode-evaluation-harness
bigcode-project/bigcode-evaluation-harness
1,049Auf GitHub ansehenA framework for the evaluation of autoregressive code generation language models.
Standardized framework for evaluating autoregressive code generation models.
Python - abacaj/code-eval
abacaj/code-eval
This is a repo I use to run human-eval on code models, adjust as needed. Some scripts were adjusted from wizardcoder repo (process_eval.py). The evaluation code is duplicated in several files, mostly to handle edge cases around model tokenizing and loading (will clean it up).
Lightweight framework for evaluating code models on HumanEval benchmarks.
Python
