2 repository-uri
Awesome GitHub RepositoriesDecoder-Only Inference
Execution modes optimized for models that only use a decoder for autoregressive generation.
Distinct from Sequence Decoders: Focuses on general text generation using decoder-only architectures, whereas the parent is a general component logic.
Explore 2 awesome GitHub repositories matching artificial intelligence & ml · Decoder-Only Inference. Refine with filters or upvote what's useful.
Awesome Decoder-Only Inference GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- google/gemma_pytorch
google/gemma_pytorch
5,697Vezi pe GitHubThe official PyTorch implementation of Google's Gemma models
Loads and executes a decoder-only transformer to generate text completions from a prompt on CPU, GPU, or TPU.
Pythongemmagooglepytorch - opennmt/ctranslate2
OpenNMT/CTranslate2
4,319Vezi pe GitHubCTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and GPU with optimizations for speed and memory efficiency. It functions as a model format converter, quantization tool, and REST API server, enabling deployment of neural machine translation, automatic speech recognition, and text generation models. The engine distinguishes itself through a suite of runtime optimizations including layer fusion, weight-matrix quantization, batch-by-length grouping, and a caching allocator that reuses GPU memory. It supports tensor-parallel model di
CTranslate2 runs text generation using decoder-only Transformer models converted from OpenNMT-py format.
C++avxavx2cpp
