Awesome GitHub RepositoriesCommand Line Model Inferences

TensorFlow.js is a JavaScript machine learning library used for training and deploying models in web browsers and server-side environments. It functions as a browser-based model trainer, a WebAssembly inference engine, and a WebGPU accelerated tensor library for low-level linear algebra. The project also includes a model converter to transform Python-based models into optimized formats for JavaScript execution. The library distinguishes itself through a pluggable backend architecture that allows mathematical operations to be executed via CPU, WebGL, or WebGPU. It supports the conversion of Py

The simplest way to run LLaMA on your local machine

Mistral Inference is a library for running Mistral large language models on a GPU, generating text from prompts with token streaming. It loads pretrained model weights from local disk or a remote registry into GPU memory, then produces output tokens one by one for real-time display in interactive applications. The library supports multimodal prompts that accept image URLs alongside text, enabling visual description and reasoning. It includes content safety guardrails that scan generated text against predefined policies to block or flag policy violations. For structured interactions, it provid

该项目是一个大语言模型推理库和框架，旨在运行用于文本生成、问题解决和编码辅助的模型。它包括一个用于处理图像和文本组合输入的多模态框架，以及一个基于模型推理执行外部工具的工具调用实现。 该系统具有分布式 GPU 推理引擎，可将大型模型工作负载分散到多个图形处理器上，以提高处理速度并满足内存需求。它还通过预打包的镜像和依赖项提供容器化模型部署，以便在隔离环境中运行推理引擎。 该库涵盖了一系列功能，包括多模态输入分析、函数调用集成，以及用于预测缺失代码段的“中间填充”（fill-in-the-middle）编码。它还支持通过命令行界面进行交互式模型聊天，以维持对话会话。

WhisperLiveKit is a real-time speech-to-text server that transcribes streaming audio into text with ultra-low latency using Whisper models. It serves transcription capabilities through REST endpoints and WebSocket connections, enabling external applications to send audio and receive transcriptions as words are spoken, making it suitable for live captioning or voice interfaces. The project distinguishes itself by combining real-time transcription with speaker diarization, assigning transcribed words to individual speakers during live audio streams for meeting or interview transcripts. It also

Cog is a machine learning packaging tool and containerized model wrapper that bundles models and their dependencies into standardized Docker containers. It functions as an environment manager and inference server, ensuring consistent model execution across different hardware systems by resolving GPU drivers, system libraries, and Python dependencies. The project distinguishes itself by automatically generating RESTful HTTP servers and OpenAPI schemas based on defined model input and output types. It manages large model weights as external fixtures to optimize image size and utilizes a slot-ba

PrusaSlicer is a G-code generator that converts 3D models into machine instructions for FFF and mSLA printers, handling slicing, infill, and support generation. It provides a command-line slicing interface for processing models and profiles via terminal commands without a graphical user interface, and includes a G-code customization engine that inserts user-defined macros, variables, and post-processing scripts into generated G-code for tailored machine control. The software also manages multi-material prints by coordinating multiple extruders and filament colors, assigning materials to model

Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning on Intel CPUs, GPUs, and NPUs. It provides a distributed inference engine for scaling models across multiple accelerators, a multimodal model runtime for vision and speech tasks, and a low-bit model quantization tool for converting weights into INT4, FP8, and GGUF formats. The project features a parameter-efficient finetuning framework that enables model adaptation using QLoRA and DPO on Intel hardware. It distinguishes itself by providing specialized optimizations for Intel XP

Vowpal Wabbit is an open-source machine learning system designed for online learning, where models update incrementally from streaming data without requiring full retraining. It provides a reduction-based learning framework that composes complex tasks from simpler algorithms, and includes a feature hashing trick that maps unbounded feature names into a fixed-size vector space to keep memory usage constant regardless of dataset size. The system supports distributed training across a cluster using an allreduce protocol for synchronized updates, and offers an active learning query strategy that s

Metaseq 是一个 Transformer 序列建模工具包，专为使用开放预训练权重进行序列到序列模型的训练、微调和部署而设计。它为大语言模型训练提供了一个全面的框架，包括用于序列数据集处理的专用工具，以及用于通过 API 请求生成文本的独立推理服务器。 该项目具有用于模型量化的专用实用程序，可将参数精度降低到 8 位，从而降低内存使用量并提高推理速度。它还包括一个检查点转换流水线，将模型权重转换为针对高性能推理引擎优化的结构。 该框架通过使用张量并行和分片数据并行，支持跨 GPU 集群的大规模训练。其他功能涵盖 NLP 数据集准备、用于迁移学习的预训练权重加载，以及用于进度可视化的训练指标跟踪。

The TensorFlow Cookbook is a collection of code examples and recipes for building, training, and deploying machine learning models using TensorFlow. It covers the full model lifecycle, from constructing neural networks and training them with configurable parameters to packaging trained models for production deployment with unit tests and multi-device support. The project also integrates TensorBoard for logging and visualizing computational graphs, scalar summaries, and histograms during training. The cookbook demonstrates a wide range of machine learning techniques, including convolutional ne

该项目是一个用于本地大语言模型推理的库和命令行界面。它支持从各种模型架构生成文本补全和聊天响应。 该项目提供用于权重量化的工具以减小内存占用，并通过 GPU 卸载结合硬件加速以提高计算速度。它还包括用于通过测量特定数据集上的困惑度来评估模型的实用程序。 能力涵盖完整的推理生命周期，包括二进制模型加载、基于模板的提示词结构化以及用于保持对话上下文的会话持久化。它还支持任务编排，允许将多个模型调用序列化为流水线以进行多步操作。

SAHI 是一个切片推理框架和计算机视觉流水线，旨在检测高分辨率图像中的小物体。它提供了一个将大图像划分为重叠补丁的系统，以防止标准模型缩小时通常发生的细节丢失，并附带图像平铺工具和 COCO 数据集工具包。 该项目的独特之处在于提供了一个模型无关的预测封装器，将不同的机器学习框架标准化为统一接口。这使得它能够在保持一致输出格式的同时，跨各种模型后端实现切片推理和目标检测。 除了推理之外，该框架还涵盖了 COCO 和 YOLO 格式的数据集管理，包括用于标注图像切片、类别重映射和数据集合并的工具。它还包括一个用于模型性能评估和监控的套件，具有精度和召回率指标计算、检测错误分析以及结果可视化功能。 该工具集可通过命令行接口访问，用于自动化跨图像目录和视频流的推理工作流。

该项目是一个容器化的开发栈和应用程序框架，用于构建检索增强生成（RAG）系统。它提供了一个 Docker 化的 AI 沙盒，集成了本地模型运行时、知识图谱和向量存储，以实现上下文感知聊天机器人的创建。 该栈的特点在于其基于图的向量存储，它结合了结构化知识图谱与向量索引，用于语义和结构化数据检索。它允许在 CPU 或 GPU 加速下进行本地模型托管，从而无需依赖外部云 API 即可执行生成任务。 该框架涵盖了广泛的功能，包括 PDF 文档的处理和索引、基于容器的 AI 服务编排，以及实现基于事实的响应生成。它包含一个带有增量响应流的 Web 聊天界面，以及用于在不同语言模型提供商之间切换的标准接口。 该环境通过容器编排进行引导，以快速部署预配置的模型和数据库栈。

xtuner 是一个用于大语言模型的综合训练引擎，提供用于预训练、监督微调以及视觉-语言多模态模型优化的工具包。它作为一个分布式训练加速器和专门的框架，用于扩展专家混合（MoE）模型，并通过人类反馈强化学习（RLHF）来对齐模型行为。 该项目的特色在于先进的内存和计算优化，例如用于超长上下文窗口的序列并行，以及用于减少 GPU 空闲时间的交错流水线并行。它提供了一套专门的偏好优化套件，实现了如组相对策略优化（GRPO）和直接偏好优化（DPO）等技术，以优化模型策略和奖励系统。 广泛的功能领域涵盖跨多节点的分布式模型训练、多模态数据集准备以及基于适配器（Adapter）的微调管理。该引擎还包括用于模型评估、权重合并以及将训练参数导出到推理引擎的工具。 训练通过标准化的配置文件和分布式启动器进行管理，以确保跨计算集群的一致结果。

MedicalGPT is an open-source framework for fine-tuning large language models, with a dedicated focus on adapting general models to the medical domain. It provides a complete pipeline that covers continued pretraining on domain-specific corpora, supervised instruction tuning, tokenizer vocabulary extension with medical terminology, and alignment to clinician preferences through direct preference optimization, reinforcement learning, or knowledge distillation. The framework also supports training models to invoke external tools and functions in multi-turn clinical conversations. The platform di

Aigcpanel is a visual workflow automation tool and model lifecycle manager designed for generative AI media pipelines. It provides a unified interface to install, launch, and configure both local and remote AI model endpoints, acting as an orchestration platform for large language models and AI tools. The system features a drag-and-drop node editor for chaining AI models and scripts into automated processing pipelines. It distinguishes itself with a breakpoint-aware execution model that allows users to pause and resume long media tasks from specific points in the workflow. Additionally, it in

ExecuTorch is a lightweight C++ runtime for deploying PyTorch models on mobile, embedded, and edge hardware. It provides an ahead-of-time compilation pipeline that exports, quantizes, and lowers model graphs into compact serialized programs, then executes them through a minimal runtime with hardware acceleration and on-device large language model inference capabilities. The project distinguishes itself through a hardware accelerator delegate system that partitions model subgraphs and offloads computation to specialized backends including NPUs, GPUs, and DSPs from Apple, Arm, Intel, MediaTek,

Distributed-llama is a distributed inference engine and command line tool for running large language models across multiple networked machines. It functions as a compute cluster manager that coordinates worker nodes to share the computational load of a single model. The system utilizes tensor parallelism to shard model weights across different hosts, allowing the execution of models that exceed the memory capacity of a single piece of hardware. It includes a dedicated format converter to transform standard model files into a compatible binary layout optimized for distributed loading. The eng

Foundry-Local 是一款机器学习开发工具，旨在促进私有的、设备端推理和模型管理。它提供了一个本地服务器环境，直接在用户的硬件上托管机器学习模型，确保所有数据处理（包括提示处理和音频转录）都保留在本地环境中，而无需外部云连接。 该项目通过自动化整个模型生命周期（包括资产的发现、下载和版本控制以保持与主机硬件的兼容性）而脱颖而出。它具有一个硬件抽象层，可自动检测并选择最高效的可用处理器进行计算密集型任务，从而无需手动配置即可实现硬件加速执行。 除了核心推理外，该工具还包括一个用于交互式模型探索和性能验证的命令行界面。它还提供标准化的 API 代理，使用行业标准协议将传入请求映射到本地模型端点，以支持与外部软件框架的集成。