LLM Council is a framework for orchestrating multi-model workflows that generates consensus-based responses by querying multiple language models simultaneously. It functions as a multi-model orchestrator that distributes user prompts across various endpoints, aggregates the resulting outputs, and synthesizes them into a single, unified final answer through a designated chairman model.
The system distinguishes itself by implementing an anonymized peer review loop, which masks model identities during the evaluation phase to ensure that critiques and rankings are based solely on output quality rather than brand bias. This process allows models to critique one another, facilitating objective performance assessment and comparative analysis within a structured deliberation pipeline.
The framework includes comprehensive capabilities for workflow auditing and system resilience. It provides transparent audit trails that expose raw model outputs and intermediate ranking data, allowing users to verify the logic behind complex decision-making. Additionally, the architecture supports resilient partial failure handling, ensuring that the deliberation process continues using only successful model responses if individual components encounter errors or timeouts.
