How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali
The main features of michaelfeil/infinity are: Inference and Serving, Model Serving & Deployment, Model Serving Engines.
Projects with overlapping indexed features include: vllm-project/vllm — vLLM is a high-throughput inference engine designed for the efficient serving and execution of large language models.… ggerganov/llama.cpp — llama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across… berriai/litellm — LiteLLM is a unified gateway and proxy server designed to centralize access to over one hundred language model… bentoml/openllm — OpenLLM is a framework for deploying, managing, and scaling open-source large language models. flowiseai/flowise — Flowise is a low-code platform designed for building and deploying complex language model workflows through a visual,… ggml-org/llama.cpp — Llama.cpp is an inference engine designed for the local execution of text-based and multimodal language models on…
vLLM is a high-throughput inference engine designed for the efficient serving and execution of large language models. It functions as a production-ready distributed model server, providing standard API protocols for online serving while also supporting offline batch processing. The system is built to maximize token generation speed and memory efficiency, enabling both large-scale cloud deployments and local execution on personal hardware. The project distinguishes itself through advanced memory management and request scheduling techniques, most notably its use of non-contiguous key-value cach
LiteLLM is a unified gateway and proxy server designed to centralize access to over one hundred language model providers. It provides a standardized API interface that abstracts vendor-specific schemas, allowing developers to interact with diverse models through a single, consistent format. By acting as a central traffic management layer, it enables organizations to route, secure, and govern model interactions across multiple deployments. The platform distinguishes itself through its policy-driven architecture, which uses configuration-based routing to manage traffic distribution, load balanc
OpenLLM is a framework for deploying, managing, and scaling open-source large language models
Flowise is a low-code platform designed for building and deploying complex language model workflows through a visual, node-based interface. It functions as an orchestrator for autonomous multi-agent systems, allowing users to construct conversational pipelines by connecting language models, memory stores, and external tools on a drag-and-drop canvas. The platform distinguishes itself through its support for sophisticated agentic patterns, including supervisor-worker delegation and iterative reasoning strategies. Users can design directed acyclic graphs to manage conditional branching, state p