How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
LMFlow is a comprehensive suite for large language model fine-tuning, context extension, multimodal processing, and inference execution. It provides a toolkit for updating model parameters through full tuning or memory-efficient adapter algorithms, alongside an inference engine for executing tuned models via command-line or web-based interfaces. The framework includes a dedicated alignment suite for supervised tuning and reward model training to refine model behavior. It features a context window extender to increase maximum input lengths and a multimodal framework for building chatbots that
FastChat is a training and serving platform for large language models that provides an integrated toolkit for fine-tuning, hosting, and benchmarking chatbots. It functions as an inference server capable of hosting multiple models and exposing them via a standardized API for chat applications. The platform distinguishes itself through a distributed model controller that manages worker nodes and routes requests across a hardware-agnostic inference layer supporting various accelerators. It includes a dedicated evaluation framework for assessing model quality using automated judges, multi-turn di
This project is a library and command-line interface for local large language model inference. It enables the generation of text completions and chat responses from various model architectures. The project provides tools for weight quantization to reduce memory footprints and incorporates hardware acceleration through GPU offloading to increase computation speed. It also includes utilities for model evaluation by measuring perplexity on specific datasets. Capabilities cover the full inference lifecycle, including binary model loading, template-based prompt structuring, and session persistenc
jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti
llama-rs is a local large language model inference engine implemented in Rust. It enables the execution of model computations on local hardware to generate text responses from user prompts.
The main features of setzer22/llama-rs are: Inference Execution, Command Line Inference Interfaces, Memory-Mapped Weight Loaders, Local Inference Engines, Model Quantization Frameworks, Weight Quantization, High-Performance Tensor Libraries, Command-Line.
Open-source alternatives to setzer22/llama-rs include: optimalscale/lmflow — LMFlow is a comprehensive suite for large language model fine-tuning, context extension, multimodal processing, and… lm-sys/fastchat — FastChat is a training and serving platform for large language models that provides an integrated toolkit for… rustformers/llm — This project is a library and command-line interface for local large language model inference. It enables the… dusty-nv/jetson-inference — jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU… infrasys-ai/aisystem — AISystem is a comprehensive AI full-stack infrastructure project covering the entire pipeline from AI chip… meta-llama/llama-models — This project provides a foundational framework and reference implementation for executing causal language modeling and…