SciSharp/LLamaSharp
LLamaSharp
LLamaSharp is a .NET LLM inference library and local runtime that enables the execution of large language models on CPU and GPU hardware. It serves as a multimodal AI library capable of processing both text and image inputs to generate analytical textual responses without relying on external APIs.
The project distinguishes itself as a grammar-based text generator that enforces specific output formats, such as JSON, through constrained sampling pipelines. It also functions as a retrieval augmented generation framework integration, allowing the combination of local inference with external data indexing.
The library covers a broad range of capabilities including text embedding generation for vector search, chat session management for stateful conversations, and image analysis. It further provides tools for model weight quantization, document relevance reranking, and the offloading of model layers to graphics hardware.
Features
- Local AI Inference - Executes large language models directly on local CPU and GPU hardware for private, low-latency inference.
- Local LLM Execution - Enables the execution of large language models on local hardware with optimized tokenization and memory management.
- Retrieval-Augmented Generation - Grounds model responses by retrieving relevant documents from external data indexing for factual context.
- Hardware Acceleration Backends - Configures the runtime to use specific hardware acceleration backends like CUDA, Vulkan, or AVX.
- Inference Libraries - Provides a high-performance .NET library for loading and executing large language models on local CPU and GPU hardware.
- Grammar-Constrained Generation - Enforces structural validity of model outputs using formal grammar rules to ensure responses follow formats like JSON.
- GGUF Model Loaders - Loads and initializes quantized model weights from the GGUF binary format to minimize memory usage.
- Structured Output Generators - Forces language models to produce strictly typed, machine-readable formats like JSON using grammar constraints.
- Text Embedding Generators - Provides utilities for transforming text into vector representations to support semantic search and similarity analysis.
- Grammar-Constrained Samplers - Restricts token generation based on formal language rules to enforce strict output schemas.
- Model Layer Offloading - Distributes specific model computation layers to graphics hardware to accelerate processing speed.
- Vector Search - Implements the conversion of text into embeddings to facilitate similarity-based information retrieval and document reranking.
- GPU Hardware Acceleration - Distributes specific model computation layers to graphics hardware to accelerate processing speed.
- AI Agent Development - Provides the necessary primitives to build interactive AI agents, including conversation state and streaming responses within .NET.
- Multimodal Frameworks - Provides a framework capable of processing both text and image inputs to generate analytical textual responses.
- Vision Analysis - Generates descriptive or analytical textual responses by processing image files alongside text prompts.
- AI Integration Frameworks - Integrates the local inference engine with external software kits for orchestration and bot development.
- Chat Session Management - Maintains conversational history and state across multi-turn dialogues to provide continuity.
- Conversation Context Tracking - Stores and retrieves historical query-response pairs to maintain continuity within multi-turn dialogues.
- Document Rerankers - Scores and re-orders document lists based on their relevance to a specific user query.
- RAG Framework Integrations - Integrates local inference with external data indexing to implement retrieval-augmented generation workflows.
- Incremental Inference Streaming - Streams model outputs to clients incrementally as they are generated to reduce perceived latency.
- GPU Acceleration Configurations - Allows precise configuration of the number of model layers offloaded to the GPU to optimize inference performance.
- Sampling Controls - Adjusts the randomness and creativity of generated text using custom sampling pipelines and parameters.
- Weight Quantization - Compresses large model weights into lower-precision formats to reduce memory usage and increase inference speed.
- Artificial Intelligence - C# bindings for running LLaMA and GPT models locally.
