Awesome GitHub RepositoriesInference Speed Profiling

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

MMPose is a PyTorch-based pose estimation toolbox and deep learning training pipeline designed for detecting 2D and 3D keypoints on humans, animals, and faces. It serves as a computer vision model zoo and a framework for both 2D pose estimation and 3D pose lifting. The project is distinguished by its modular architecture and extensibility, employing a registry-based system and hierarchical configurations to allow for custom algorithm integration and model pipeline customization. It supports diverse estimation paradigms, including top-down, bottom-up, and two-stage pose lifting workflows. The

GLM-4 is a large language model and fine-tuning framework designed for human-like text production, complex reasoning, and multilingual conversation. It functions as a multimodal system capable of processing high-resolution visual content and as a long-context model designed to analyze documents with a context window of up to one million tokens. The project differentiates itself through a function calling interface that enables AI agent development by connecting the model to external APIs and real-time web browsing. It includes specialized capabilities for generating functional programming cod

LMCache is a distributed key-value cache manager and tiering system designed to accelerate large language model inference. It functions as a tiered storage layer that offloads tensors from GPU memory to CPU RAM, local disks, or remote object stores, enabling the reuse of cached prefixes across different inference sessions and serving engines. The system differentiates itself through a disaggregated prefill-decode model, which separates prompt processing from token generation by transferring caches between distributed compute nodes. It utilizes peer-to-peer orchestration to share and retrieve

Dynamo is a distributed inference orchestration platform designed for large language models. It functions as a system to coordinate prefill and decode phases across GPU nodes, utilizing a multi-backend runtime adapter to connect engines like vLLM and TensorRT-LLM through a unified block-oriented memory interface. An OpenAI-compatible API server provides the frontend for integration with existing tools and clients. The project is distinguished by its disaggregated serving architecture, which separates prompt processing and token generation onto independent GPU pools to optimize throughput and

Dieses Projekt ist ein Leistungsoptimierer und Ressourcen-Bencher für AWS Lambda. Es analysiert das Verhältnis zwischen Ausführungsgeschwindigkeit und Kosten, indem es verschiedene Speicherkonfigurationen testet, um die kosteneffizientesten Einstellungen zu identifizieren und die Betriebsausgaben zu minimieren. Das Tool nutzt einen AWS-Step-Functions-Orchestrator, um die Ausführung und Datensammlung mehrerer Funktionstestläufe über verschiedene Leistungsstufen hinweg zu automatisieren. Es simuliert Produktions-Workloads durch das Injizieren benutzerdefinierter statischer oder Remote-Daten und die Verwendung gewichteter Payload-Verteilung, um reale Verkehrsmuster nachzuahmen. Die Suite deckt mehrere Funktionsbereiche ab, einschließlich iterativer Speicherabtastung und metrikbasierter Kostenmodellierung zur Visualisierung von Leistungs-Trade-offs. Sie bietet automatisierte Ressourcenbereinigung für temporäre Funktionsversionen und Aliase, private Netzwerkkonfiguration für eingeschränkte interne Ressourcen und Remote-Payload-Laden, um Standard-Aufrufgrößenbeschränkungen zu umgehen. Die Bereitstellung erfolgt über Infrastructure-as-Code-Konstrukte, um eine konsistente Umgebungseinrichtung und Wiederholbarkeit zu gewährleisten.

LightLLM is a high-performance serving framework for deploying and executing large language models. It functions as a multi-GPU inference engine and server capable of handling dense architectures, mixture-of-experts designs, and multimodal models that process both text and images. The system is distinguished by its specialized support for Mixture-of-Experts models using expert parallelism and fused kernels. It implements structured text generation through deterministic state machines and pushdown automata to enforce precise output formats. To optimize throughput, the framework employs specula

This project is a containerized local AI infrastructure stack designed to deploy large language models and vector databases on private hardware. It functions as an orchestration platform that combines AI runners, knowledge graphs, and a visual workflow builder for creating agentic chatflows and automating tasks via tool integration. The platform distinguishes itself through a low-code approach to agent orchestration, utilizing a visual interface to design complex sequences and connect agents to external tools and search engines. It includes a dedicated local observability stack to track promp