Awesome GitHub RepositoriesLocal Inference CLI

Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

Flux is a diffusion model inference engine designed for text-to-image generation and image-to-image manipulation. It provides a system for executing open-weight models to transform natural language descriptions into visual imagery or to modify existing images. The project distinguishes itself through a flow-matching framework for image generation and a structural image controller. This controller allows for guided synthesis by using depth maps and Canny edge detection to constrain the geometry and composition of the output. The toolkit covers a broad range of image editing capabilities, incl

Mistral Inference is a library for running Mistral large language models on a GPU, generating text from prompts with token streaming. It loads pretrained model weights from local disk or a remote registry into GPU memory, then produces output tokens one by one for real-time display in interactive applications. The library supports multimodal prompts that accept image URLs alongside text, enabling visual description and reasoning. It includes content safety guardrails that scan generated text against predefined policies to block or flag policy violations. For structured interactions, it provid

InternVL is a vision-language model framework that fuses a visual encoder with a large language model to translate image features into textual tokens for reasoning. It provides a system for multimodal inference and dialogue, enabling the processing of images and text to answer questions or generate descriptions. The project is distinguished by its high-resolution image processing, which uses dynamic tiling to maintain detail for images up to 4K resolution, and its chain-of-thought visual reasoning for solving complex mathematical and spatial problems. It also supports temporal frame sampling

OpenChatKit is a training and inference toolkit for large language models. It provides a comprehensive set of tools for managing the model lifecycle, including a fine-tuning pipeline, a model weight converter, and a command-line interface for interacting with conversational agents. The toolkit features a framework for retrieval augmented generation, allowing models to incorporate relevant context from external vector indices. It also includes utilities for converting trained model checkpoints into formats compatible with standard inference libraries. The project covers conversational AI trai

FastDeploy is a high-performance deployment framework for large language models, vision models, and multimodal models. It provides the infrastructure to launch model services that process combined image, video, and text inputs, exposing these capabilities through a standardized, OpenAI-compatible API for chat and text completions. The project distinguishes itself through advanced inference pipeline engineering and GPU optimization. It employs speculative decoding, tensor parallelism, and a disaggregated execution model that separates prefill and decode phases across different hardware resourc

该项目提供了一个在本地消费级硬件上运行大语言模型并执行基于文档的问答的框架。通过将基于 CPU 的推理引擎与本地向量数据库集成，用户无需依赖云端 API 或专用 GPU 即可处理信息。 该系统作为一个命令行工具，管理私有信息处理的全生命周期。它将本地文本文件转换为可搜索的向量嵌入，使模型能够检索相关上下文，并基于用户提供的特定原始材料生成回答。通过使用量化模型执行，该框架降低了内存和计算需求，从而在标准硬件上也能流畅运行。 该项目涵盖了文档索引、语义检索和上下文感知生成的完整流水线。它通过将所有文档摄入、嵌入生成和模型推理任务保留在本地环境中，确保了数据隐私。