2 Repos
Awesome GitHub RepositoriesInference Request Pipelines
Systems that tokenize inputs and route them through inference engines.
Distinct from Inference Request APIs: Candidates focus on general API management or specific multimodal normalizers; this is the core LLM request orchestration.
Explore 2 awesome GitHub repositories matching artificial intelligence & ml · Inference Request Pipelines. Refine with filters or upvote what's useful.
Awesome Inference Request Pipelines GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- kubeflow/kfserving
kubeflow/kfserving
5,576Auf GitHub ansehenKServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines inference services as custom resources with declarative YAML specifications, enabling a Kubernetes-native approach to model deployment and lifecycle management. The platform leverages Knative-based serverless scaling for automatic scale-to-zero and revision management, and supports a pluggable serving runtime architecture that maps model formats to containerized execution environments. KServe distinguishes itself through model-aware autoscaling that scales replicas based on token
Directs traffic between predictor, transformer, and explainer components in inference pipelines.
Go - modeltc/lightllm
ModelTC/LightLLM
3,901Auf GitHub ansehenLightLLM is a high-performance serving framework for deploying and executing large language models. It functions as a multi-GPU inference engine and server capable of handling dense architectures, mixture-of-experts designs, and multimodal models that process both text and images. The system is distinguished by its specialized support for Mixture-of-Experts models using expert parallelism and fused kernels. It implements structured text generation through deterministic state machines and pushdown automata to enforce precise output formats. To optimize throughput, the framework employs specula
Tokenizes text and multimodal inputs and coordinates their delivery through a routing and inference pipeline.
Pythondeep-learninggptllama
