# karpathy/llm-council

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/karpathy-llm-council).**

14,761 stars · 2,979 forks · Python

## Links

- GitHub: https://github.com/karpathy/llm-council
- awesome-repositories: https://awesome-repositories.com/repository/karpathy-llm-council.md

## Description

LLM Council is a framework for orchestrating multi-model workflows that generates consensus-based responses by querying multiple language models simultaneously. It functions as a multi-model orchestrator that distributes user prompts across various endpoints, aggregates the resulting outputs, and synthesizes them into a single, unified final answer through a designated chairman model.

The system distinguishes itself by implementing an anonymized peer review loop, which masks model identities during the evaluation phase to ensure that critiques and rankings are based solely on output quality rather than brand bias. This process allows models to critique one another, facilitating objective performance assessment and comparative analysis within a structured deliberation pipeline.

The framework includes comprehensive capabilities for workflow auditing and system resilience. It provides transparent audit trails that expose raw model outputs and intermediate ranking data, allowing users to verify the logic behind complex decision-making. Additionally, the architecture supports resilient partial failure handling, ensuring that the deliberation process continues using only successful model responses if individual components encounter errors or timeouts.

## Tags

### Artificial Intelligence & ML

- [Multi-Model AI Orchestrators](https://awesome-repositories.com/f/artificial-intelligence-ml/multi-model-ai-orchestrators.md) — Orchestrates multiple language models to compare outputs, rank responses, and synthesize a final consensus answer.
- [Model Comparison Tools](https://awesome-repositories.com/f/artificial-intelligence-ml/model-comparison-tools.md) — Anonymizes and evaluates responses from various language models to identify the most accurate information. ([source](https://github.com/karpathy/llm-council/blob/master/main.py))
- [AI Response Synthesizers](https://awesome-repositories.com/f/artificial-intelligence-ml/ai-response-synthesizers.md) — Uses a chairman model to synthesize diverse peer responses into a single high-quality answer.
- [Evaluation Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/evaluation-frameworks.md) — Provides an automated framework for auditing deliberation processes and ensuring objective decision-making.
- [LLM Comparison Interfaces](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-evaluation-and-validation/llm-comparison-interfaces.md) — Evaluates and ranks outputs from multiple models to identify the most accurate information.
- [Model Evaluation Tools](https://awesome-repositories.com/f/artificial-intelligence-ml/model-evaluation-tools.md) — Eliminates bias in performance assessments by anonymizing peer outputs during the critique process.
- [Deliberation Audit Interfaces](https://awesome-repositories.com/f/artificial-intelligence-ml/reasoning-models/reasoning-transparency-interfaces/deliberation-audit-interfaces.md) — Exposes raw model outputs and ranking data to allow verification of complex decision-making logic.
- [Chairman-Led Synthesis](https://awesome-repositories.com/f/artificial-intelligence-ml/response-synthesis-engines/chairman-led-synthesis.md) — Uses a designated chairman model to synthesize peer-ranked outputs into a unified final response.
- [Prompt Chaining Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/prompt-chaining-frameworks.md) — Orchestrates simultaneous prompts across multiple models to gather diverse responses for comparative analysis. ([source](https://github.com/karpathy/llm-council/blob/master/main.py))

### Data & Databases

- [LLM](https://awesome-repositories.com/f/data-databases/consensus-engines/llm.md) — Uses a designated chairman model to review peer-ranked outputs and generate a unified final response.

### Software Engineering & Architecture

- [Anonymized Review Loops](https://awesome-repositories.com/f/software-engineering-architecture/peer-review-workflows/anonymized-review-loops.md) — Implements an anonymized peer review loop to ensure objective quality assessment of model outputs.
- [Consensus Tools](https://awesome-repositories.com/f/software-engineering-architecture/decision-frameworks/consensus-tools.md) — Provides a structured framework for synthesizing consensus-based responses from multiple language model outputs. ([source](https://github.com/karpathy/llm-council#readme))
- [Peer Review Workflows](https://awesome-repositories.com/f/software-engineering-architecture/peer-review-workflows.md) — Masks model identities during comparative assessments to eliminate bias and ensure objective rankings. ([source](https://github.com/karpathy/llm-council/blob/master/CLAUDE.md))
- [Fan-Out Dispatchers](https://awesome-repositories.com/f/software-engineering-architecture/request-dispatchers/fan-out-dispatchers.md) — Distributes user prompts across multiple model endpoints simultaneously to gather diverse responses.
- [Failure Handling Policies](https://awesome-repositories.com/f/software-engineering-architecture/failure-handling-policies.md) — Ensures system availability by continuing deliberation processes despite individual model component failures.
- [Resilient Deliberation Strategies](https://awesome-repositories.com/f/software-engineering-architecture/failure-handling-policies/resilient-deliberation-strategies.md) — Ensures system resilience by continuing to process requests using only successful model outputs when individual components encounter errors or timeouts. ([source](https://github.com/karpathy/llm-council/blob/master/CLAUDE.md))

### Networking & Communication

- [Model Critique Rankings](https://awesome-repositories.com/f/networking-communication/peer-to-peer-networking-extensions/model-critique-rankings.md) — Facilitates impartial peer-based ranking by prompting models to critique each other's outputs. ([source](https://github.com/karpathy/llm-council#readme))

### System Administration & Monitoring

- [Execution Audit Trails](https://awesome-repositories.com/f/system-administration-monitoring/execution-audit-trails.md) — Provides transparent audit trails of raw model outputs and rankings to help verify decision-making logic. ([source](https://github.com/karpathy/llm-council/blob/master/CLAUDE.md))
- [Audit Logs](https://awesome-repositories.com/f/system-administration-monitoring/audit-logs.md) — Reviews raw model outputs and deliberation logs to verify decision-making logic and debug performance.

### Development Tools & Productivity

- [Configuration-Driven Orchestrators](https://awesome-repositories.com/f/development-tools-productivity/configuration-driven-orchestrators.md) — Manages model lifecycles and interaction patterns using external configuration definitions.