microsoft/guidance
Guidance
Guidance is a control framework and generation orchestrator for large language models. It provides a programming layer to steer model outputs through structured templates, schema enforcement, and logical flow management.
The framework distinguishes itself by interleaving model generation with local code execution, enabling the use of loops and conditional branching within a single session. It employs grammar-based token constraints and regular expressions to force models to sample only from tokens that satisfy a specific structural format, ensuring strict adherence to predefined data models.
The project further optimizes performance by injecting known tokens directly into the output stream to bypass unnecessary model computation. It also includes a mock execution layer for debugging generation logic and validating prompts without calling external APIs.
Features
- Interleaved Logic Execution - Allows interleaving of model generation with local code execution for complex loops and conditional branching.
- Output Constraint Engines - Enforces structured output formats like JSON or specific grammars during model inference.
- Generation Flow Orchestrators - Coordinates complex sequences of model calls and conditional logic to manage generation workflows.
- Schema Enforcement Layers - Provides layers that force model outputs to strictly adhere to specific JSON schemas during generation.
- Prompt Templates - Combines static text and dynamic variables into structured templates for language model prompts.
- Schema Enforcement Tools - Ensures language model responses strictly adhere to specific data formats and structural schemas.
- Structured Output Enforcements - Ensures generated data strictly follows predefined schemas or formats for reliable machine parsing.
- Grammar-Constrained Samplers - Implements mechanisms that restrict token sampling based on formal grammars to enforce structured output.
- LLM Development Frameworks - Provides a programming layer to constrain and steer LLM outputs through structured templates and logical flow.
- Schema-Enforced Output Parsers - Constrains model responses using regular expressions and grammars to prevent hallucinations in structured fields.
- Generator-Based Flow Control - Interleaves model generation with logic like conditionals and loops to steer execution paths.
- LLM Reasoning Workflows - Orchestrates complex sequences of model calls integrated with logic, loops, and conditionals.
- Custom Generation Functions - Provides the ability to create reusable functions that build complex, consistent structural outputs with the model.
- Prompt Debugging Utilities - Enables validating generation logic and refining prompts locally with mock models before deploying to APIs.
- Generation Speed Optimizers - Improves generation speed by inserting known tokens directly into the output stream to reduce GPU passes.
- In-Stream Token Injections - Implements performance gains by injecting known tokens into the output stream to reduce GPU compute and latency.
- Inference Compute Optimizations - Reduces GPU costs and latency by inserting known tokens to bypass unnecessary model computation.
- Token Stream Injections - Optimizes performance by injecting known tokens directly into the output stream to bypass redundant model computation.
- Generation Logic Simulation - Provides a utility to validate candidate strings and test execution flows locally using a mock model.
- API Mocking - Provides a mock execution layer to simulate model responses for testing generation logic without API calls.
- Application Development - Library for interleaving generation, prompting, and logical control.
- Large Language Models - Control language model output with structured prompts.
- Large Language Models (LLMs) - Listed in the “Large Language Models (LLMs)” section of the The Incredible Pytorch awesome list.
