1 Repo
Awesome GitHub RepositoriesInference Compute Optimizations
Reducing GPU costs by bypassing model passes through token injection.
Distinct from Token Optimization Utilities: Distinct from general Token Optimization Utilities: focuses on reducing compute/GPU passes via injection rather than prompt compression.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Inference Compute Optimizations. Refine with filters or upvote what's useful.
Awesome Inference Compute Optimizations GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- microsoft/guidance
microsoft/guidance
21,502Auf GitHub ansehenGuidance is a control framework and generation orchestrator for large language models. It provides a programming layer to steer model outputs through structured templates, schema enforcement, and logical flow management. The framework distinguishes itself by interleaving model generation with local code execution, enabling the use of loops and conditional branching within a single session. It employs grammar-based token constraints and regular expressions to force models to sample only from tokens that satisfy a specific structural format, ensuring strict adherence to predefined data models.
Reduces GPU costs and latency by inserting known tokens to bypass unnecessary model computation.
Jupyter Notebook
