abetlen/llama-cpp-python
Llama Cpp Python
llama-cpp-python provides a Python interface for the llama.cpp library, enabling the execution of large language models with hardware acceleration. It functions as a GGUF model loader and a structured text generator capable of running inference servers and multimodal runtimes for processing both text and image inputs.
The project distinguishes itself through a local inference server that exposes model capabilities via an OpenAI-compatible web API. It supports advanced execution techniques including speculative decoding, weight quantization, and layer-based GPU offloading to manage memory across system RAM and VRAM.
The library covers a broad range of AI capabilities, including text completion, embedding generation, and the enforcement of structured outputs via JSON schemas or formal grammars. It also provides infrastructure for tool use through external function calling and manages model extensions via LoRA adapter injection.
Users can fetch model files directly from Hugging Face and maintain model state persistence for resuming generation.
Features
- LLM Python Bindings - Provides the primary Python interface for the llama.cpp library to run hardware-accelerated models.
- Chat Completion Services - Exposes an API for generating conversational responses using structured message sequences.
- Embedding Generators - Provides utilities for transforming text into numerical vector representations for semantic search and clustering.
- Local Model Serving - Hosts a local API server that allows external applications to communicate with a hosted model.
- Inference API Servers - Hosts a local inference server that exposes model capabilities through standardized web APIs.
- Model Loading - Imports model weights from local paths and manages layer offloading to the GPU.
- Model Loaders - Implements a GGUF model loader with support for GPU offloading and adapter injection.
- Model Quantization - Implements weight quantization to reduce the memory footprint and increase the inference speed of GGUF models.
- Precision Quantization - Supports weight quantization to reduce memory usage and increase inference speed on consumer hardware.
- Text Completion Engines - Produces text completions based on prompts using configurable sampling parameters and stop sequences.
- Text Generation APIs - Provides an interface for interacting with language models to produce text, chat, or code completions.
- Local LLM Tools - Enables running large language models on local hardware for text generation and embeddings.
- OpenAI-Compatible Servers - Implements the OpenAI API specification to ensure compatibility with existing third-party AI clients.
- Chat Template Management - Provides tools for defining and managing the structured templates used to prompt conversational models.
- Chat Template Formatters - Implements methods for converting chat history into model-specific token sequences for consistent input.
- Output Constraint Engines - Enforces structured output formats like JSON during model inference to ensure valid data generation.
- OpenAI-Compatible Inference Servers - Provides a local inference server that exposes model capabilities through a standardized OpenAI-compatible API.
- Speculative Decoding Strategies - Implements speculative decoding using a small draft model to accelerate text generation speed.
- LoRA Adapter Loaders - Applies low-rank adaptation weights to a base model during runtime to modify behavior.
- Multimodal Processing - Enables the simultaneous processing and integration of text and image inputs using vision-capable models.
- Multimodal Vision Interfaces - Provides a runtime environment for processing both text and image inputs via multimodal vision interfaces.
- Structured Output Generators - Constrains model output to valid JSON schemas or formal grammars for structured data generation.
- Grammar-Constrained Samplers - Restricts token generation based on formal language rules to enforce specific output schemas.
- Text Tokenizers - Transforms raw UTF-8 strings into integer tokens required for model processing.
- Token Decoders - Converts internal token representations back into human-readable text for final display.
- Tool-Calling Schemas - Enables external function calling by enforcing structured parameter schemas for tool integration.
- Model Layer Offloading - Distributes model layers between system RAM and VRAM to enable execution of large models on limited hardware.
- Tool Use and Function Calling - Enables language models to interact with external APIs and software tools through structured function calling.
- GPU Acceleration - Provides hardware acceleration on macOS by offloading compute-intensive model operations to the GPU via Metal.
- Python-C Interfaces - Maps high-level Python function calls to optimized low-level C++ memory operations for model execution.
- Inference Engines - Python interface for the llama.cpp inference engine.
