abetlen/llama-cpp-python

Features

• LLM Python Bindings - Provides the primary Python interface for the llama.cpp library to run hardware-accelerated models.
• Chat Completion Services - Exposes an API for generating conversational responses using structured message sequences.
• Embedding Generators - Provides utilities for transforming text into numerical vector representations for semantic search and clustering.
• Local Model Serving - Hosts a local API server that allows external applications to communicate with a hosted model.
• Inference API Servers - Hosts a local inference server that exposes model capabilities through standardized web APIs.
• Model Loading - Imports model weights from local paths and manages layer offloading to the GPU.
• Model Loaders - Implements a GGUF model loader with support for GPU offloading and adapter injection.
• Model Quantization - Implements weight quantization to reduce the memory footprint and increase the inference speed of GGUF models.
• Precision Quantization - Supports weight quantization to reduce memory usage and increase inference speed on consumer hardware.
• Text Completion Engines - Produces text completions based on prompts using configurable sampling parameters and stop sequences.
• Text Generation APIs - Provides an interface for interacting with language models to produce text, chat, or code completions.
• Local LLM Tools - Enables running large language models on local hardware for text generation and embeddings.
• OpenAI-Compatible Servers - Implements the OpenAI API specification to ensure compatibility with existing third-party AI clients.
• Chat Template Management - Provides tools for defining and managing the structured templates used to prompt conversational models.
• Chat Template Formatters - Implements methods for converting chat history into model-specific token sequences for consistent input.
• Output Constraint Engines - Enforces structured output formats like JSON during model inference to ensure valid data generation.
• OpenAI-Compatible Inference Servers - Provides a local inference server that exposes model capabilities through a standardized OpenAI-compatible API.
• Speculative Decoding Strategies - Implements speculative decoding using a small draft model to accelerate text generation speed.
• LoRA Adapter Loaders - Applies low-rank adaptation weights to a base model during runtime to modify behavior.
• Multimodal Processing - Enables the simultaneous processing and integration of text and image inputs using vision-capable models.
• Multimodal Vision Interfaces - Provides a runtime environment for processing both text and image inputs via multimodal vision interfaces.
• Structured Output Generators - Constrains model output to valid JSON schemas or formal grammars for structured data generation.
• Grammar-Constrained Samplers - Restricts token generation based on formal language rules to enforce specific output schemas.
• Text Tokenizers - Transforms raw UTF-8 strings into integer tokens required for model processing.
• Token Decoders - Converts internal token representations back into human-readable text for final display.
• Tool-Calling Schemas - Enables external function calling by enforcing structured parameter schemas for tool integration.
• Model Layer Offloading - Distributes model layers between system RAM and VRAM to enable execution of large models on limited hardware.
• Tool Use and Function Calling - Enables language models to interact with external APIs and software tools through structured function calling.
• GPU Acceleration - Provides hardware acceleration on macOS by offloading compute-intensive model operations to the GPU via Metal.
• Python-C Interfaces - Maps high-level Python function calls to optimized low-level C++ memory operations for model execution.
• Inference Engines - Python interface for the llama.cpp inference engine.

Star history