How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
Evaluating and reproducing real-world robot manipulation policies (e.g., RT-1, RT-1-X, Octo) in simulation under common setups (e.g., Google Robot, WidowX+Bridge) (CoRL 2024)
The main features of simpler-env/simplerenv are: Model Evaluation and Benchmarking, Simulators.
Projects with overlapping indexed features include: open-mmlab/mmsegmentation — MMSegmentation is an open-source semantic segmentation toolbox built on PyTorch that provides a modular, configurable… packtpublishing/llm-engineers-handbook — This project is an educational resource and engineering guide for building, deploying, and optimizing large language… facebookresearch/parlai — ParlAI is a conversational AI research framework designed for training, evaluating, and sharing dialogue models using… infrasys-ai/aiinfra. openai/simple-evals — This project is a language model evaluation framework and benchmarking tool designed to measure the accuracy and… facebookresearch/slowfast — SlowFast is a PyTorch video understanding framework and spatiotemporal neural network library. It serves as a toolset…
MMSegmentation is an open-source semantic segmentation toolbox built on PyTorch that provides a modular, configurable framework for building, training, evaluating, and deploying segmentation models. At its core, it offers a config-driven pipeline that assembles training, evaluation, and inference workflows by parsing hierarchical configuration files, with a modular component registry that enables plug-and-play composition of neural network modules, optimizers, datasets, and metrics. The framework supports the full model lifecycle through a unified runner interface that controls training, testi
ParlAI is a conversational AI research framework designed for training, evaluating, and sharing dialogue models using a unified interface for datasets and agents. It functions as a PyTorch-based training platform and a dialogue data collection system, providing a centralized model zoo for the distribution of versioned pretrained agents. The project distinguishes itself through a knowledge-grounded retrieval system that combines dense and sparse indexing to ground responses in external information. It also provides a comprehensive infrastructure for gathering human-AI interaction data via inte
This project is a language model evaluation framework and benchmarking tool designed to measure the accuracy and performance of models across diverse datasets. It provides a system for implementing model-based graders, running standardized tests for mathematical reasoning, coding, and factuality, and calculating quantified performance metrics such as precision, recall, F1 scores, and pass-at-k. The framework utilizes model-based grading and rubrics to validate response quality against expert-defined criteria. It includes a multi-model benchmarking loop and a model-agnostic API interface to co
