2 repositorios
Awesome GitHub RepositoriesThroughput Optimizations
Techniques specifically targeting the increase of inference throughput using attention kernels and batching.
Distinct from Inference Performance Optimizers: Focuses on throughput-increasing mechanisms like paged KV caching and in-flight batching, rather than general compression/quantization.
Explore 2 awesome GitHub repositories matching artificial intelligence & ml · Throughput Optimizations. Refine with filters or upvote what's useful.
Awesome Throughput Optimizations GitHub Repositories
Encuentra los mejores repositorios con IA.Buscaremos los repositorios que mejor coincidan usando IA.
- dusty-nv/jetson-inference
dusty-nv/jetson-inference
8,734Ver en GitHubjetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti
Increases inference throughput using custom attention kernels, in-flight batching, and paged KV caching.
C++caffecomputer-visiondeep-learning - rlinf/rlinf
RLinf/RLinf
2,502Ver en GitHubRLinf is a distributed reinforcement learning orchestrator and embodied AI training framework. It provides the infrastructure to train vision-language-action models and robotic policies using a combination of reinforcement learning and supervised fine-tuning. The system is designed for scaling workloads across GPU clusters, managing the placement of actors, rollout workers, and environment components. It features a specialized robotics data collection pipeline for gathering teleoperated demonstrations and simulation trajectories into standardized replay buffers, alongside a hardware interface
Maximizes simulator throughput by overlapping model forward passes with environment stepping across vectorized slices.
Pythonagentic-aiembodied-aireinforcement-learning
