fchollet/ARC-AGI
ARC AGI
This project is a standardized set of abstraction and reasoning problems designed for benchmarking the ability of artificial intelligence models to learn new rules. It functions as a fluid intelligence test and a reasoning benchmark, utilizing a collection of grid-based puzzles and a program synthesis dataset to evaluate how agents generate algorithms from examples.
The project focuses on measuring general fluid intelligence and the capacity for zero-shot generalization, testing whether a system can apply learned logic to unseen problems without relying on task-specific training. It provides a framework for abstract reasoning research, specifically evaluating program synthesis and the ability of models to discover latent patterns for novel grid transformations.
The system incorporates a grid-based domain representation and a combinatorial search space of geometric and topological operations. It includes a human-in-the-loop interface that allows for the manual construction of output grids to define ground truth for validation and benchmarking.
Features
- Fluid Intelligence Benchmarks - Serves as a standardized benchmark to measure general fluid intelligence and program synthesis in AI models.
- Geometric Knowledge Priors - Utilizes a core set of innate geometric biases such as symmetry and containment to guide the program search.
- Example-Based Synthesis - Derives general transformation rules by learning patterns from a small set of input-output examples.
- Fluid Intelligence Tests - Features a series of logic puzzles requiring the discovery of latent patterns to solve grid transformations.
- LLM Benchmarking - Provides a dataset of grid-based tasks to measure general fluid intelligence and reasoning in LLMs.
- AI Evaluation Frameworks - Provides a standardized set of abstraction and reasoning problems to assess the quality of AI reasoning.
- Problem Domain Representations - Represents reasoning tasks as multi-dimensional arrays of integers mapping to colors and spatial coordinates.
- Zero-Shot Generalization - Tests the ability of AI systems to solve novel reasoning tasks without training on the specific problem distribution.
- Abstract Reasoning Research - Provides a framework for researching how machines perform conceptual abstraction and general reasoning.
- Program Synthesis - Tests the capacity of models to generate executable code that solves abstract logic problems from few examples.
- Datasets - Includes a collection of visual reasoning puzzles designed to test an agent's capacity to generate algorithms.
- Standardized Benchmarks - Uses a standardized set of grid-based puzzles to benchmark an AI system's fluid intelligence.
- Agent Generalization Testing - Evaluates whether AI can apply learned logic to unseen problems without task-specific training.
- Human-in-the-Loop Interfaces - Provides an interface for manual grid manipulation to define ground truth and validate reasoning tasks.
- Combinatorial Search Spaces - Implements a search space of primitive geometric and topological operations to discover valid transformation programs.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
