Awesome GitHub RepositoriesInference Speed Profiling

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

MMPose is a PyTorch-based pose estimation toolbox and deep learning training pipeline designed for detecting 2D and 3D keypoints on humans, animals, and faces. It serves as a computer vision model zoo and a framework for both 2D pose estimation and 3D pose lifting. The project is distinguished by its modular architecture and extensibility, employing a registry-based system and hierarchical configurations to allow for custom algorithm integration and model pipeline customization. It supports diverse estimation paradigms, including top-down, bottom-up, and two-stage pose lifting workflows. The

GLM-4 is a large language model and fine-tuning framework designed for human-like text production, complex reasoning, and multilingual conversation. It functions as a multimodal system capable of processing high-resolution visual content and as a long-context model designed to analyze documents with a context window of up to one million tokens. The project differentiates itself through a function calling interface that enables AI agent development by connecting the model to external APIs and real-time web browsing. It includes specialized capabilities for generating functional programming cod

LMCache is a distributed key-value cache manager and tiering system designed to accelerate large language model inference. It functions as a tiered storage layer that offloads tensors from GPU memory to CPU RAM, local disks, or remote object stores, enabling the reuse of cached prefixes across different inference sessions and serving engines. The system differentiates itself through a disaggregated prefill-decode model, which separates prompt processing from token generation by transferring caches between distributed compute nodes. It utilizes peer-to-peer orchestration to share and retrieve

Dynamo is a distributed inference orchestration platform designed for large language models. It functions as a system to coordinate prefill and decode phases across GPU nodes, utilizing a multi-backend runtime adapter to connect engines like vLLM and TensorRT-LLM through a unified block-oriented memory interface. An OpenAI-compatible API server provides the frontend for integration with existing tools and clients. The project is distinguished by its disaggregated serving architecture, which separates prompt processing and token generation onto independent GPU pools to optimize throughput and

هذا المشروع عبارة عن أداة تحسين أداء ومقياس موارد لـ AWS Lambda. يحلل المقايضة بين سرعة التنفيذ والتكلفة عن طريق اختبار إعدادات ذاكرة مختلفة لتحديد الإعدادات الأكثر فعالية من حيث التكلفة وتقليل الإنفاق التشغيلي. تستخدم الأداة منسق AWS Step Functions لأتمتة تنفيذ وجمع بيانات عمليات اختبار الوظائف المتعددة عبر مستويات طاقة مختلفة. تحاكي أحمال عمل الإنتاج عن طريق حقن بيانات ثابتة أو بعيدة مخصصة واستخدام توزيع حمولة مرجح لمحاكاة أنماط حركة المرور الواقعية. تغطي المجموعة عدة مجالات قدرة، بما في ذلك أخذ عينات الذاكرة التكرارية ونمذجة التكلفة القائمة على المقاييس لتصور مقايضات الأداء. توفر تنظيفاً آلياً للموارد لإصدارات الوظائف المؤقتة والأسماء المستعارة، وإعداد شبكة خاصة للموارد الداخلية المقيدة، وتحميل حمولة بعيد لتجاوز حدود حجم الاستدعاء القياسية. يتم التعامل مع النشر عبر بنيات البنية التحتية ككود (infrastructure-as-code) لضمان إعداد بيئة متسق وقابلية التكرار.

LightLLM is a high-performance serving framework for deploying and executing large language models. It functions as a multi-GPU inference engine and server capable of handling dense architectures, mixture-of-experts designs, and multimodal models that process both text and images. The system is distinguished by its specialized support for Mixture-of-Experts models using expert parallelism and fused kernels. It implements structured text generation through deterministic state machines and pushdown automata to enforce precise output formats. To optimize throughput, the framework employs specula

This project is a containerized local AI infrastructure stack designed to deploy large language models and vector databases on private hardware. It functions as an orchestration platform that combines AI runners, knowledge graphs, and a visual workflow builder for creating agentic chatflows and automating tasks via tool integration. The platform distinguishes itself through a low-code approach to agent orchestration, utilizing a visual interface to design complex sequences and connect agents to external tools and search engines. It includes a dedicated local observability stack to track promp