1 个仓库
Awesome GitHub RepositoriesInference Compute Optimizations
Reducing GPU costs by bypassing model passes through token injection.
Distinct from Token Optimization Utilities: Distinct from general Token Optimization Utilities: focuses on reducing compute/GPU passes via injection rather than prompt compression.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Inference Compute Optimizations. Refine with filters or upvote what's useful.
Awesome Inference Compute Optimizations GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- microsoft/guidance
microsoft/guidance
21,502在 GitHub 上查看Guidance is a control framework and generation orchestrator for large language models. It provides a programming layer to steer model outputs through structured templates, schema enforcement, and logical flow management. The framework distinguishes itself by interleaving model generation with local code execution, enabling the use of loops and conditional branching within a single session. It employs grammar-based token constraints and regular expressions to force models to sample only from tokens that satisfy a specific structural format, ensuring strict adherence to predefined data models.
Reduces GPU costs and latency by inserting known tokens to bypass unnecessary model computation.
Jupyter Notebook
