Awesome GitHub RepositoriesCommand Line Model Inferences

TensorFlow.js is a JavaScript machine learning library used for training and deploying models in web browsers and server-side environments. It functions as a browser-based model trainer, a WebAssembly inference engine, and a WebGPU accelerated tensor library for low-level linear algebra. The project also includes a model converter to transform Python-based models into optimized formats for JavaScript execution. The library distinguishes itself through a pluggable backend architecture that allows mathematical operations to be executed via CPU, WebGL, or WebGPU. It supports the conversion of Py

The simplest way to run LLaMA on your local machine

Mistral Inference is a library for running Mistral large language models on a GPU, generating text from prompts with token streaming. It loads pretrained model weights from local disk or a remote registry into GPU memory, then produces output tokens one by one for real-time display in interactive applications. The library supports multimodal prompts that accept image URLs alongside text, enabling visual description and reasoning. It includes content safety guardrails that scan generated text against predefined policies to block or flag policy violations. For structured interactions, it provid

This project is a large language model inference library and framework designed to run models for text generation, problem solving, and coding assistance. It includes a multimodal framework for processing combined image and text inputs and a tool-use implementation that enables the execution of external functions based on model reasoning. The system features a distributed GPU inference engine that spreads large model workloads across multiple graphics processors to increase processing speed and meet memory requirements. It also provides containerized model deployment through pre-packaged imag

WhisperLiveKit is a real-time speech-to-text server that transcribes streaming audio into text with ultra-low latency using Whisper models. It serves transcription capabilities through REST endpoints and WebSocket connections, enabling external applications to send audio and receive transcriptions as words are spoken, making it suitable for live captioning or voice interfaces. The project distinguishes itself by combining real-time transcription with speaker diarization, assigning transcribed words to individual speakers during live audio streams for meeting or interview transcripts. It also

Cog is a machine learning packaging tool and containerized model wrapper that bundles models and their dependencies into standardized Docker containers. It functions as an environment manager and inference server, ensuring consistent model execution across different hardware systems by resolving GPU drivers, system libraries, and Python dependencies. The project distinguishes itself by automatically generating RESTful HTTP servers and OpenAPI schemas based on defined model input and output types. It manages large model weights as external fixtures to optimize image size and utilizes a slot-ba

PrusaSlicer is a G-code generator that converts 3D models into machine instructions for FFF and mSLA printers, handling slicing, infill, and support generation. It provides a command-line slicing interface for processing models and profiles via terminal commands without a graphical user interface, and includes a G-code customization engine that inserts user-defined macros, variables, and post-processing scripts into generated G-code for tailored machine control. The software also manages multi-material prints by coordinating multiple extruders and filament colors, assigning materials to model

Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning on Intel CPUs, GPUs, and NPUs. It provides a distributed inference engine for scaling models across multiple accelerators, a multimodal model runtime for vision and speech tasks, and a low-bit model quantization tool for converting weights into INT4, FP8, and GGUF formats. The project features a parameter-efficient finetuning framework that enables model adaptation using QLoRA and DPO on Intel hardware. It distinguishes itself by providing specialized optimizations for Intel XP

Vowpal Wabbit is an open-source machine learning system designed for online learning, where models update incrementally from streaming data without requiring full retraining. It provides a reduction-based learning framework that composes complex tasks from simpler algorithms, and includes a feature hashing trick that maps unbounded feature names into a fixed-size vector space to keep memory usage constant regardless of dataset size. The system supports distributed training across a cluster using an allreduce protocol for synchronized updates, and offers an active learning query strategy that s

Metaseq एक ट्रांसफॉर्मर सीक्वेंस मॉडलिंग टूलकिट है जिसे ओपन प्री-ट्रेंड वेट्स का उपयोग करके सीक्वेंस-टू-सीक्वेंस मॉडल को प्रशिक्षित करने, फाइन-ट्यून करने और डिप्लॉय करने के लिए डिज़ाइन किया गया है। यह बड़े भाषा मॉडल प्रशिक्षण के लिए एक व्यापक फ्रेमवर्क प्रदान करता है, जिसमें सीक्वेंस डेटासेट प्रोसेसिंग के लिए समर्पित टूल्स और API अनुरोधों के माध्यम से टेक्स्ट उत्पन्न करने के लिए एक स्टैंडअलोन इन्फरेंस सर्वर शामिल है। इस प्रोजेक्ट में पैरामीटर परिशुद्धता को आठ बिट्स तक कम करने के लिए मॉडल क्वांटिज़ेशन के लिए विशेष उपयोगिताएँ शामिल हैं, जो मेमोरी उपयोग को कम करती हैं और इन्फरेंस गति को बढ़ाती हैं। इसमें मॉडल वेट्स को उच्च-प्रदर्शन इन्फरेंस इंजन के लिए अनुकूलित संरचनाओं में बदलने के लिए एक चेकपॉइंट रूपांतरण पाइपलाइन भी शामिल है। फ्रेमवर्क टेंसर पैरेललिज़्म और शार्ड डेटा पैरेललिज़्म के उपयोग के माध्यम से GPU क्लस्टर में बड़े पैमाने पर प्रशिक्षण का समर्थन करता है। अतिरिक्त क्षमताओं में NLP डेटासेट तैयारी, ट्रांसफर लर्निंग के लिए प्री-ट्रेंड वेट लोडिंग, और प्रगति विज़ुअलाइज़ेशन के लिए प्रशिक्षण मेट्रिक्स ट्रैकिंग शामिल है।

The TensorFlow Cookbook is a collection of code examples and recipes for building, training, and deploying machine learning models using TensorFlow. It covers the full model lifecycle, from constructing neural networks and training them with configurable parameters to packaging trained models for production deployment with unit tests and multi-device support. The project also integrates TensorBoard for logging and visualizing computational graphs, scalar summaries, and histograms during training. The cookbook demonstrates a wide range of machine learning techniques, including convolutional ne

यह प्रोजेक्ट स्थानीय लार्ज लैंग्वेज मॉडल इन्फ़्रेंस के लिए एक लाइब्रेरी और कमांड-लाइन इंटरफ़ेस है। यह विभिन्न मॉडल आर्किटेक्चर से टेक्स्ट कंप्लीशन और चैट रिस्पॉन्स के जनरेशन को सक्षम बनाता है। प्रोजेक्ट मेमोरी फ़ुटप्रिंट को कम करने के लिए वेट क्वांटाइज़ेशन के लिए टूल्स प्रदान करता है और कंप्यूटेशन गति बढ़ाने के लिए GPU ऑफ़लोडिंग के माध्यम से हार्डवेयर एक्सेलेरेशन को शामिल करता है। इसमें विशिष्ट डेटासेट पर परप्लेक्सिटी को मापकर मॉडल मूल्यांकन के लिए यूटिलिटीज भी शामिल हैं। क्षमताएं पूर्ण इन्फ़्रेंस लाइफसाइकिल को कवर करती हैं, जिसमें बाइनरी मॉडल लोडिंग, टेम्पलेट-आधारित प्रॉम्प्ट स्ट्रक्चरिंग और कन्वर्सेशनल कॉन्टेक्स्ट बनाए रखने के लिए सेशन पर्सिस्टेंस शामिल है। यह टास्क ऑर्केस्ट्रेशन का भी समर्थन करता है, जिससे कई मॉडल कॉल्स को मल्टी-स्टेप ऑपरेशंस के लिए पाइपलाइन में अनुक्रमित किया जा सकता है।

SAHI is a sliced inference framework and computer vision pipeline designed to detect small objects in high-resolution images. It provides a system for dividing large images into overlapping patches to prevent the detail loss that typically occurs during standard model downscaling, alongside an image tiling utility and a COCO dataset toolkit. The project distinguishes itself by offering a model-agnostic prediction wrapper that standardizes different machine learning frameworks into a unified interface. This allows it to implement sliced inference and object detection across various model backe

This project is a containerized development stack and application framework for building retrieval-augmented generation systems. It provides a dockerized AI sandbox that integrates local model runtimes, knowledge graphs, and vector stores to enable the creation of contextual chatbots. The stack is distinguished by its graph-based vector store, which combines structured knowledge graphs with vector indices for both semantic and structural data retrieval. It allows for local model hosting with CPU or GPU acceleration, enabling generative tasks without reliance on external cloud APIs. The frame

xtuner बड़े भाषा मॉडल के लिए एक व्यापक प्रशिक्षण इंजन है, जो प्री-ट्रेनिंग, सुपरवाइज्ड फाइन-ट्यूनिंग और विज़न-लैंग्वेज मल्टीमॉडल मॉडल के अनुकूलन के लिए एक टूलकिट प्रदान करता है। यह एक वितरित प्रशिक्षण त्वरक और Mixture-of-Experts मॉडल को स्केल करने और मानव फीडबैक से सुदृढीकरण शिक्षण के माध्यम से मॉडल व्यवहार को संरेखित करने के लिए एक विशेष फ्रेमवर्क के रूप में कार्य करता है। प्रोजेक्ट उन्नत मेमोरी और कंप्यूट अनुकूलन के माध्यम से खुद को अलग करता है, जैसे अल्ट्रा-लॉन्ग कॉन्टेक्स्ट विंडो के लिए सीक्वेंस पैरेललिज्म और GPU आइडल समय को कम करने के लिए इंटरलीव्ड पाइपलाइन पैरेललिज्म। यह प्राथमिकता अनुकूलन के लिए एक समर्पित सूट प्रदान करता है, जो मॉडल नीतियों और इनाम प्रणालियों को परिष्कृत करने के लिए Group Relative Policy Optimization और Direct Preference Optimization जैसी तकनीकों को लागू करता है। व्यापक क्षमता क्षेत्र कई नोड्स में वितरित मॉडल प्रशिक्षण, मल्टीमॉडल डेटासेट तैयारी और एडाप्टर-आधारित फाइन-ट्यूनिंग के प्रबंधन को कवर करते हैं। इंजन में मॉडल मूल्यांकन, वेट मर्जिंग और प्रशिक्षित मापदंडों को इन्फरेंस इंजन में निर्यात करने के लिए टूल भी शामिल हैं। प्रशिक्षण का प्रबंधन मानकीकृत कॉन्फ़िगरेशन फाइलों और वितरित लॉन्चरों के माध्यम से किया जाता है ताकि कंप्यूटिंग क्लस्टर में सुसंगत परिणाम सुनिश्चित किए जा सकें।

MedicalGPT is an open-source framework for fine-tuning large language models, with a dedicated focus on adapting general models to the medical domain. It provides a complete pipeline that covers continued pretraining on domain-specific corpora, supervised instruction tuning, tokenizer vocabulary extension with medical terminology, and alignment to clinician preferences through direct preference optimization, reinforcement learning, or knowledge distillation. The framework also supports training models to invoke external tools and functions in multi-turn clinical conversations. The platform di

Aigcpanel is a visual workflow automation tool and model lifecycle manager designed for generative AI media pipelines. It provides a unified interface to install, launch, and configure both local and remote AI model endpoints, acting as an orchestration platform for large language models and AI tools. The system features a drag-and-drop node editor for chaining AI models and scripts into automated processing pipelines. It distinguishes itself with a breakpoint-aware execution model that allows users to pause and resume long media tasks from specific points in the workflow. Additionally, it in

ExecuTorch is a lightweight C++ runtime for deploying PyTorch models on mobile, embedded, and edge hardware. It provides an ahead-of-time compilation pipeline that exports, quantizes, and lowers model graphs into compact serialized programs, then executes them through a minimal runtime with hardware acceleration and on-device large language model inference capabilities. The project distinguishes itself through a hardware accelerator delegate system that partitions model subgraphs and offloads computation to specialized backends including NPUs, GPUs, and DSPs from Apple, Arm, Intel, MediaTek,

Distributed-llama is a distributed inference engine and command line tool for running large language models across multiple networked machines. It functions as a compute cluster manager that coordinates worker nodes to share the computational load of a single model. The system utilizes tensor parallelism to shard model weights across different hosts, allowing the execution of models that exceed the memory capacity of a single piece of hardware. It includes a dedicated format converter to transform standard model files into a compatible binary layout optimized for distributed loading. The eng

Foundry-Local is a machine learning development tool designed to facilitate private, on-device inference and model management. It provides a local server environment that hosts machine learning models directly on the user's hardware, ensuring that all data processing, including prompt handling and audio transcription, remains within the local environment without requiring external cloud connectivity. The project distinguishes itself by automating the entire model lifecycle, including the discovery, downloading, and versioning of assets to maintain compatibility with host hardware. It features