Projects sharing features with PCA EVAL

[Website](http://craftjarvis-jarvis1.github.io/) [Paper](https://arxiv.org/abs/2311.05997) [Twitter](https://twitter.com/jeasinema/status/1723900032653643796)

UQLM: Uncertainty Quantification for Language Models, is a Python package for UQ-based LLM hallucination detection

This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks.

This project is a standardized framework for benchmarking large language models across a wide range of academic and reasoning datasets. It provides a platform for executing automated evaluation tasks to measure model accuracy and performance, ensuring consistent assessment through a structured configuration schema. The framework distinguishes itself by incorporating a dedicated utility for data decontamination, which identifies and removes overlapping training samples from evaluation sets to prevent data leakage. It also features a flexible task builder that allows users to define custom benc

An Embodied Generalist Agent in 3D World

Rigourous evaluation of LLM-synthesized code - NeurIPS 2023 & COLM 2024

Ragas is an evaluation framework and performance benchmark designed to quantify the quality of retrieval augmented generation pipelines. It functions as an application optimizer to identify bottlenecks in language model workflows using automated metrics and model-based scoring. The framework includes a system for generating synthetic datasets that mimic production scenarios and edge cases to create realistic test cases. It enables reference-free assessment, allowing the evaluation of response quality by analyzing grounding in the provided context without requiring gold-standard labels. The s

Assess, Guard, and Monitor Your LLM Applications Built by Future AGI | Docs | Platform

Giskard is an evaluation framework, testing library, and quality monitoring system for large language models and AI agents. It serves as a toolkit for quantifying model performance and reliability, providing specialized capabilities for validating retrieval-augmented generation pipelines. The project distinguishes itself through an automated red teaming tool and security scanner designed to identify vulnerabilities, prompt injections, and safety risks. It utilizes adversarial probing and synthetic edge case generation to quantify model robustness and detect information disclosure. The platfo

Wenlong Huang 1 , Chen Wang 1 , Ruohan Zhang 1 , Yunzhu Li 1,2 , Jiajun Wu 1 , Li Fei-Fei 1

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.

Lighteval is an open-source framework for running standardized benchmarks and custom evaluation tasks against language models. It provides a system for defining new evaluation tasks with custom prompts, metrics, and scoring in YAML configuration files, and integrates with the Hugging Face Hub for storing and comparing results. The framework supports evaluating models across multiple inference backends, including transformers, vllm, and custom APIs, through a unified generation and log-probability interface. It includes a pluggable metric registry for built-in and custom scoring, a prediction

An open-source visual programming environment for battle-testing prompts to LLMs.

Deliver safe & effective language models

Agentic applications give an LLM freedom over control flow in order to solve problems. While this freedom can be extremely powerful, the black box nature of LLMs can make it difficult to understand how changes in one part of your agent will affect others downstream. This makes evaluating your…

Langfuse is an open-source observability and evaluation platform designed for language model applications. It provides a centralized system for tracking execution traces, monitoring performance metrics, and managing prompt templates. By capturing hierarchical units of work and telemetry data, the platform enables developers to debug complex application lifecycles and analyze token usage, latency, and model interactions in production environments. The platform distinguishes itself through an integrated evaluation framework that allows for systematic benchmarking and automated scoring of model

FastChat is a training and serving platform for large language models that provides an integrated toolkit for fine-tuning, hosting, and benchmarking chatbots. It functions as an inference server capable of hosting multiple models and exposing them via a standardized API for chat applications. The platform distinguishes itself through a distributed model controller that manages worker nodes and routes requests across a hardware-agnostic inference layer supporting various accelerators. It includes a dedicated evaluation framework for assessing model quality using automated judges, multi-turn di

A unified evaluation framework for large language models

Voyager is an autonomous embodied agent and lifelong learning framework that uses a large language model to explore virtual environments. It functions as a code-based action controller, translating natural language instructions into executable scripts to interact with its surroundings. The system features an automatic curriculum generator that creates sequences of exploration goals to discover new items and behaviors without human intervention. It maintains a skill library manager that stores learned behaviors as reusable code fragments, which can be composed to execute complex tasks. The fr

The official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".

[Paper](https://arxiv.org/abs/2210.05663) [Website](https://mahis.life/clip-fields/) [Code](https://github.com/notmahi/clip-fields) [Data](https://osf.io/famgv) [Video](https://youtu.be/bKu7GvRiSQU)

Project webpage · Documentation (gitbooks) · Paper