eth-sri/lmql
Lmql
LMQL is a programming language and probabilistic interface that blends algorithmic logic with stochastic text generation. It functions as a constraint-guided prompting framework and structured output generator, allowing users to force model responses to adhere to strict formatting and data types.
The system distinguishes itself as an inference optimizer that increases token throughput and reduces latency. This is achieved through specialized execution strategies, including tree-based prompt caching and asynchronous batch processing.
The project covers a broad range of generation control capabilities, such as logit-based token masking, beam-search selection, and interleaved logic-prompt execution. It also provides infrastructure for real-time response streaming via WebSockets and server-sent events.
Features
- Constraint-Based Prompting - Provides a framework for forcing LLM responses to follow strict formatting and data types via logit masking.
- Real-time Generation Constraints - Forces model responses to adhere to strict formats and data types using real-time token-level constraints and logit masking.
- Interleaved Logic Execution - Merges traditional programming control flow with stochastic text generation within a single execution runtime.
- Inference Optimizations - Reduces latency and increases throughput using specialized caching and asynchronous batching for model requests.
- Structured Output Generators - Forces language models to produce strictly typed, machine-readable data formats using guided prompting.
- Prefix Caching - Stores and reuses common prompt prefix computations to avoid redundant calculations across multiple variations.
- Logit Masks - Applies binary logit masks to the output distribution to disable specific tokens and enforce data type constraints.
- Constraint-Guided Decoders - Interleaves programmatic logic with token sampling to ensure generated text adheres to strict structural requirements.
- Probabilistic Programming Languages - Integrates traditional control flow with the probabilistic nature of large language model generation.
- LLM - Blends algorithmic logic with stochastic text generation to create a language for guiding and constraining LLM outputs.
- Programmatic Workflow Logic - Combines developer-written programmatic flow with LLM queries to control the reasoning and structure of generated output.
- Algorithmic - Reduces token usage and response times through speculative execution, constraint short-circuiting, and tree-based caching.
- LLM Response Streaming - Delivers model-generated text incrementally to clients via WebSockets or server-sent events for lower perceived latency.
- Beam Search Implementations - Implements beam search algorithms to track multiple probable token sequences for more accurate model responses.
- Decoding Strategy Implementations - Implements advanced token selection algorithms like beam search to optimize the quality and accuracy of generated text.
- Query Batching - Groups multiple computational queries into single execution calls to increase token throughput and reduce latency.
- Inference - Reduces token computation time through the combined use of tree-based caching and constraint short-circuiting.
- Batch Request Processing - Executes multiple requests in parallel using an asynchronous system to increase total processing throughput.
- Inference Short-Circuiting - Bypasses unnecessary model computations when constraints make specific token paths redundant or impossible.
- Application Frameworks - Programming language for large language models.
- Prompt Optimization Frameworks - Query language for programming complex prompt logic.
- Prompting Frameworks - Query language for programming and controlling language models.
- Structured Output Generation - Language for constraint-guided and efficient model programming.
Star history
How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
