How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
Habitat-Lab is an open-source platform for training and evaluating embodied AI agents in photorealistic 3D indoor environments. It functions as a high-performance 3D indoor environment simulator that supports physics-based interaction, enabling research into navigation and manipulation tasks. The platform provides a modular task-environment abstraction that separates task logic from environment simulation, using configuration-driven pipeline assembly to compose simulation and training pipelines. It includes a hierarchical sensor-actuator architecture for mixing and matching perception and act
Deepeval is a framework for testing and evaluating large language model applications. It provides a suite of tools for executing automated regression tests, validating model output quality against defined standards, and tracing the execution of complex agent workflows. By integrating these capabilities into development pipelines, the platform ensures consistent performance and reliability throughout the software lifecycle. The platform distinguishes itself through its focus on programmatic validation and observability. It utilizes secondary language models to score output quality and employs
[Website](http://craftjarvis-jarvis1.github.io/) [Paper](https://arxiv.org/abs/2311.05997) [Twitter](https://twitter.com/jeasinema/status/1723900032653643796)
Opik is an observability and evaluation platform designed for generative AI applications and agentic workflows. It provides a centralized environment for tracing execution flows, managing prompt templates, and monitoring production performance, allowing teams to gain visibility into complex model interactions and tool usage without requiring manual application code changes. The platform distinguishes itself through its integrated approach to the AI development lifecycle, combining distributed trace instrumentation with automated evaluation frameworks. It supports model-as-a-judge scoring, syn
The main features of pkunlp-icler/pca-eval are: Embodied Agents, Evaluation Frameworks, Multimodal Perception.
Projects with overlapping indexed features include: facebookresearch/habitat-lab — Habitat-Lab is an open-source platform for training and evaluating embodied AI agents in photorealistic 3D indoor… confident-ai/deepeval — Deepeval is a framework for testing and evaluating large language model applications. It provides a suite of tools for… craftjarvis/jarvis-1 — [[Website]](http://craftjarvis-jarvis1.github.io/) [[Paper]](https://arxiv.org/abs/2311.05997)… cvs-health/uqlm — UQLM: Uncertainty Quantification for Language Models, is a Python package for UQ-based LLM hallucination detection. declare-lab/instruct-eval — This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on… comet-ml/opik — Opik is an observability and evaluation platform designed for generative AI applications and agentic workflows. It…