Lightning-AI/LitServe
3,894 stele·292 fork-uri·Python·Apache-2.0·10 vizualizărilightning.ai/litserve?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme↗
LitServe
LitServe este un framework Python pentru servere de inferență AI și un framework de servire LLM conceput pentru inferență de înaltă concurență. Funcționează ca un server de model AI distribuit și un motor de inferență cu batching dinamic, oferind instrumentele necesare pentru a construi și găzdui servere personalizate care rulează modele AI.
Framework-ul se distinge printr-o coadă de cereri cu batching dinamic care grupează cererile individuale de inferență în tensori unici pentru a maximiza throughput-ul GPU. Suportă scalarea GPU distribuită, permițând sarcinilor de lucru ale modelelor să fie distribuite pe mai multe acceleratoare hardware pentru a echilibra sarcinile de calcul și a crește capacitatea totală.
Sistemul oferă o interfață wrapper de nivel înalt care decuplează preprocesarea și postprocesarea cererilor de logica de bază de execuție a modelului. Include, de asemenea, capabilități pentru streaming-ul modelelor în timp real pentru a livra output-uri incremental și utilizează o buclă de evenimente asincronă pentru a gestiona cererile de rețea concurente.
Features
- Dynamic Inference Batching - Implements a dynamic-batching request queue to maximize GPU throughput by grouping individual requests.
- AI Model Orchestration - Allows defining specific logic for how data is routed and processed through AI models.
- AI Workflow Orchestration - Provides capabilities to define custom orchestration logic for routing and processing data through AI models.
- Dynamic Batching Engines - Groups individual real-time requests into batches to maximize GPU throughput.
- Distributed Inference Scaling - Distributes model workloads across multiple hardware nodes to increase throughput and balance resources.
- Inference Servers - Offers a dedicated server framework for hosting and executing machine learning models.
- Inference Server Frameworks - Provides a high-level wrapper interface that encapsulates server boilerplate for rapid deployment of custom inference logic.
- Distributed Model Servers - Distributes model workloads across multiple accelerators to handle high-concurrency inference requests.
- Incremental Inference Streaming - Enables the incremental delivery of model outputs to clients to reduce perceived latency.
- Serving Frameworks - Provides a high-throughput deployment and serving framework specifically for large language models.
- Model Response Streaming - Ships a generator that sends model outputs incrementally to the client.
- Inference Servers - Functions as a network service that hosts AI models and provides predictions via a Python API.
- Real-Time Text Streaming - Sends AI generated responses to the user incrementally as they are produced.
- Inference Batching - Groups multiple incoming AI requests into single batches to maximize GPU hardware utilization.
- Multi-GPU Workload Distribution - Spreads inference tasks across multiple hardware accelerators to balance compute loads.
- Request Batching - Combines multiple individual requests into a single execution to optimize hardware utilization.
- Logic Decoupling - Decouples request preprocessing and postprocessing from the core model execution logic.
- Asyncio Coroutine Execution - Uses an asynchronous event loop to handle concurrent network requests without blocking execution.
- Inference and Serving - Fast serving engine with support for batching and autoscaling.
- Local Development and Serving - Deploy high-speed GPU inference with autoscaling capabilities.
- Model Serving - Flexible and easy-to-use serving engine for AI models.
- Model Serving & Deployment - Provides a flexible serving engine built on FastAPI.
- Model Serving Frameworks - Lightweight, high-performance serving engine for AI models.
- PyTorch Ecosystem - Model serving framework for PyTorch.
