2 repositorios
Awesome GitHub RepositoriesInference Stream Multiplexing
Concurrent processing of multiple inference requests using separate host threads and queues while sharing weight memory.
Distinct from Multi-Threaded Request Handling: Focuses on sharing model weight memory across multiple inference streams, unlike general network request handling.
Explore 2 awesome GitHub repositories matching software engineering & architecture · Inference Stream Multiplexing. Refine with filters or upvote what's useful.
Awesome Inference Stream Multiplexing GitHub Repositories
Encuentra los mejores repositorios con IA.Buscaremos los repositorios que mejor coincidan usando IA.
- openvinotoolkit/openvino
openvinotoolkit/openvino
10,414Ver en GitHubOpenVINO is an AI inference engine and model serving platform designed to execute optimized deep learning models across CPUs, GPUs, and NPUs through a unified API. It includes a model optimization toolkit for converting, quantizing, and compressing models from various frameworks, alongside a specialized generative AI runtime for large language models. The project distinguishes itself through a plugin-based hardware acceleration layer that maps neural network operations to vendor-specific drivers. It features advanced execution mechanisms such as continuous batching, speculative decoding, and
Processes inference requests simultaneously across multiple host threads while efficiently sharing model weight memory.
C++aicomputer-visiondeep-learning - koljab/realtimestt
KoljaB/RealtimeSTT
9,477Ver en GitHubRealtimeSTT is a local speech-to-text engine and real-time automatic speech recognition server. It utilizes transformer-based recognition and omnilingual pipelines to convert live audio streams into text, providing a WebSocket-based streaming API for raw PCM audio transmission. The project is distinguished by a dual-backend transcription pipeline that uses a lightweight engine for immediate partial suggestions and a heavier model for final high-accuracy results. It includes a wake word detection system to trigger recording and employs a shared-resource inference model to distribute heavy spee
Manages multiple concurrent user sessions by isolating audio buffers while sharing model weight memory for inference.
Pythonpythonrealtimespeech-to-text
