Awesome GitHub RepositoriesConstrained Decoding

Ludwig is a multimodal machine learning platform and low-code framework designed for building, training, and deploying neural networks. It enables the construction of models that process text, images, audio, and tabular data through a unified interface using declarative configuration files rather than custom code. The system features a specialized low-code framework for large language models, supporting supervised fine-tuning, preference alignment, and a constrained decoding tool to force structured data output via logit extraction. It also includes an automated model architecture search to i

mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe

LiteRT-LM est un framework d'inférence haute performance conçu pour exécuter des grands modèles de langage localement sur du matériel mobile, desktop et IoT. Il sert de runtime de modèle sur appareil qui utilise l'accélération CPU, GPU et NPU pour fournir un traitement à faible latence. Le framework se distingue par sa capacité à traiter des entrées texte, vision et audio via un moteur d'inférence multi-modal unique. Il dispose d'un serveur HTTP local qui émule des endpoints d'API compatibles avec OpenAI et un runtime basé sur WebGPU pour exécuter des modèles directement dans un navigateur web. Pour garantir la fiabilité des sorties, il inclut un générateur de texte contraint qui impose des schémas JSON ou des règles grammaticales sur les réponses du modèle. Le projet offre de larges capacités pour la gestion de conversations stateful, le décodage spéculatif pour augmenter les vitesses de génération de jetons, et une interface d'appel d'outils qui mappe les requêtes du modèle vers des fonctions externes. Il inclut également une intégration spécialisée pour l'écosystème Apple et un plugin dédié pour exécuter des modèles dans Flutter. Les utilisateurs peuvent exécuter des modèles via une interface en ligne de commande ou les intégrer dans des applications via des API natives.

This project is a retrieval augmented generation framework designed to build pipelines that connect unstructured data and knowledge graphs with large language models. It functions as a vector database orchestrator for indexing text and multimodal content, as well as a system for translating natural language queries into structured database commands. The framework integrates a hybrid retrieval engine that combines dense vector search with sparse keyword matching to increase the precision of retrieved contexts. It further enhances reasoning and relationship mapping through a graph-augmented ret

LightLLM is a high-performance serving framework for deploying and executing large language models. It functions as a multi-GPU inference engine and server capable of handling dense architectures, mixture-of-experts designs, and multimodal models that process both text and images. The system is distinguished by its specialized support for Mixture-of-Experts models using expert parallelism and fused kernels. It implements structured text generation through deterministic state machines and pushdown automata to enforce precise output formats. To optimize throughput, the framework employs specula