awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
pytorch-labs avatar

pytorch-labs/gpt-fast

0
View on GitHub↗

Gpt Fast

gpt-fast is a PyTorch transformer inference engine designed for low-latency text generation. It functions as a distributed GPU inference library, a quantized model runner, and a speculative decoding framework.

The system utilizes a speculative decoding workflow where a small draft model predicts token sequences for verification by a larger model to accelerate generation. It supports quantized model execution to reduce memory footprint and implements tensor parallelism to split computations across multiple GPUs.

The project includes a standardized evaluation harness to measure the accuracy and performance of transformer models. It manages inference efficiency through key-value cache management and the use of native PyTorch tensor operations.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Features

  • Generative Text Inference - Functions as a low-latency text generation engine using native PyTorch operations.
  • Speculative Decoding Strategies - Provides a speculative decoding engine that predicts multiple tokens using a draft model for faster generation.
  • PyTorch Tensor Operations - Uses direct PyTorch tensor operations to minimize overhead in the autoregressive generation loop.
  • Weight Quantization - Supports weight-based quantization to reduce the memory footprint and latency of transformer models.
  • Tensor Parallelism - Implements tensor parallelism to split model weights and computations across multiple GPUs.
  • Transformer Inference Engines - Provides a high-performance inference engine optimized specifically for transformer models.
  • Speculative Decoding - Implements a speculative decoding framework to accelerate generation via draft-model predictions.
  • Distributed Inference Frameworks - Provides a framework for distributing transformer model inference across multiple GPUs.
  • KV Cache Management - Implements key-value cache management to avoid redundant computations during sequential token generation.
  • Multi-GPU Inference Runtimes - Acts as a multi-GPU inference runtime that utilizes tensor parallelism for large transformer models.
  • Quantized Model Runners - Provides a runtime designed to execute transformer models with quantized weights.
  • Large Language Models - High-performance inference implementation for transformer models.
  • Large Language Models (LLMs) - Listed in the “Large Language Models (LLMs)” section of the The Incredible Pytorch awesome list.
6,225 stars·573 forks·Python·BSD-3-Clause·25 views

Star history

Star history chart for pytorch-labs/gpt-fastStar history chart for pytorch-labs/gpt-fast

How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

Frequently asked questions

What does pytorch-labs/gpt-fast do?

gpt-fast is a PyTorch transformer inference engine designed for low-latency text generation. It functions as a distributed GPU inference library, a quantized model runner, and a speculative decoding framework.

What are the main features of pytorch-labs/gpt-fast?

The main features of pytorch-labs/gpt-fast are: Generative Text Inference, Speculative Decoding Strategies, PyTorch Tensor Operations, Weight Quantization, Tensor Parallelism, Transformer Inference Engines, Speculative Decoding, Distributed Inference Frameworks.

Which projects share features with pytorch-labs/gpt-fast?

Projects with overlapping indexed features include: meta-pytorch/gpt-fast — gpt-fast is a PyTorch transformer inference engine designed for text generation using a native tensor library… mistralai/mistral-src — This project is a large language model inference library and framework designed to run models for text generation,… intel/ipex-llm — Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning… opennmt/ctranslate2 — CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and… nvidia/fastertransformer — FasterTransformer is a high-performance inference optimization library and distributed runtime designed to accelerate… facebookresearch/metaseq — Metaseq is a transformer sequence modeling toolkit designed for training, fine-tuning, and deploying…

Projects sharing features with Gpt Fast

These projects share indexed features with Gpt Fast. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
  • meta-pytorch/gpt-fastmeta-pytorch avatar

    meta-pytorch/gpt-fast

    6,223View on GitHub↗

    gpt-fast is a PyTorch transformer inference engine designed for text generation using a native tensor library implementation. It provides a runtime for executing large language models without the need for external C++ extensions. The project implements speculative decoding to accelerate generation by using a small draft model for token prediction and a larger model for verification. It further optimizes performance through a compiled prefill stage and a multi-GPU tensor parallelism library that shards linear layers across multiple graphics processing units. Memory efficiency is managed throu

    Python
    View on GitHub↗6,223
  • mistralai/mistral-srcmistralai avatar

    mistralai/mistral-src

    10,821View on GitHub↗

    This project is a large language model inference library and framework designed to run models for text generation, problem solving, and coding assistance. It includes a multimodal framework for processing combined image and text inputs and a tool-use implementation that enables the execution of external functions based on model reasoning. The system features a distributed GPU inference engine that spreads large model workloads across multiple graphics processors to increase processing speed and meet memory requirements. It also provides containerized model deployment through pre-packaged imag

    Jupyter Notebook
    View on GitHub↗10,821
  • intel/ipex-llmintel avatar

    intel/ipex-llm

    8,836View on GitHub↗

    Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning on Intel CPUs, GPUs, and NPUs. It provides a distributed inference engine for scaling models across multiple accelerators, a multimodal model runtime for vision and speech tasks, and a low-bit model quantization tool for converting weights into INT4, FP8, and GGUF formats. The project features a parameter-efficient finetuning framework that enables model adaptation using QLoRA and DPO on Intel hardware. It distinguishes itself by providing specialized optimizations for Intel XP

    Python
    View on GitHub↗8,836
  • opennmt/ctranslate2OpenNMT avatar

    OpenNMT/CTranslate2

    4,319View on GitHub↗

    CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and GPU with optimizations for speed and memory efficiency. It functions as a model format converter, quantization tool, and REST API server, enabling deployment of neural machine translation, automatic speech recognition, and text generation models. The engine distinguishes itself through a suite of runtime optimizations including layer fusion, weight-matrix quantization, batch-by-length grouping, and a caching allocator that reuses GPU memory. It supports tensor-parallel model di

    C++avxavx2cpp
    View on GitHub↗4,319
Compare all 30 related projects→