2 repository-uri
Awesome GitHub RepositoriesMulti-Turn Serving Optimizations
Specialized serving techniques for stateful multi-turn interactions, such as session-affinity routing and prefix caching.
Distinct from Optimized Model Serving: Focuses on stateful multi-turn interaction efficiency specifically, whereas Optimized Model Serving is general high-performance serving.
Explore 2 awesome GitHub repositories matching artificial intelligence & ml · Multi-Turn Serving Optimizations. Refine with filters or upvote what's useful.
Awesome Multi-Turn Serving Optimizations GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- yangjianxin1/firefly
yangjianxin1/Firefly
6,642Vezi pe GitHubFirefly is a training framework and inference engine for large language models. It functions as a toolkit for pre-training and fine-tuning various open-weight architectures, providing a system for model alignment and parameter-efficient fine-tuning. The project includes utilities for merging adapter weights back into base models to create standalone files. It also provides a model alignment toolkit to format training data according to specific prompt templates, ensuring conversational consistency across different models. The framework supports distributed model training and preference-based
Provides optimized serving for stateful multi-turn interactions using memory-efficient inference.
Pythonalpacaaquilabaichuan - thudm/slime
THUDM/slime
4,259Vezi pe GitHubSLIME is a distributed reinforcement learning framework for large language model post-training that bridges Megatron training with SGLang inference servers. It orchestrates scalable RL loops across GPU clusters, decoupling training and inference into independent processes that communicate over HTTP and NCCL for independent scaling and fault tolerance. The system supports multi-agent reinforcement learning workflows with parallel agent instances, customizable rollout strategies, and personalized agent serving that improves models from prior conversations without disrupting API serving. The fra
Implements session-affinity routing to reuse prefix caches across multi-turn interactions.
Python
