Open-source alternatives to GPTCache

Mastra is an orchestration framework designed for building, deploying, and managing autonomous AI agents and multi-agent systems. It provides a comprehensive suite of primitives for creating resilient AI applications, including durable workflow orchestration, event-driven agent loops, and semantic memory management. By integrating these core components, the platform enables developers to build complex, multi-step processes that can reason about goals and execute tasks without manual intervention. The framework distinguishes itself through its focus on observability and secure, isolated execut

llmware is a Python framework for AI agent orchestration and model management, designed to coordinate multi-model workflows and autonomous agents. It provides a unified model catalog and standardized interface to execute specialized language models for complex research, analysis, and structured data generation. The project distinguishes itself through its heavy emphasis on local execution and quantized inference, allowing models to run on private infrastructure using CPU, GPU, and NPU acceleration via runtimes like ONNX and OpenVino. It features a specialized ability to translate natural lang

Higress is an AI API gateway and cloud-native traffic manager that functions as a Kubernetes ingress controller. It provides a centralized system for routing, securing, and optimizing traffic directed toward large language models, AI agents, and microservice architectures. The project distinguishes itself through deep AI orchestration, including the ability to host and manage Model Context Protocol servers that transform REST APIs into tools for AI agents. It features specialized AI infrastructure for model request proxying, protocol translation across multiple providers, and semantic-based c

Chonkie is a text chunking library designed for retrieval-augmented generation pipelines. It functions as a semantic text splitter and RAG ingestion pipeline, transforming raw text into embedded segments for storage in vector databases. The project distinguishes itself through specialized splitting strategies, including an AST-based code splitter for preserving logical boundaries in source code and a semantic text splitter that uses embedding models to determine boundaries based on meaning. It also provides a vector database ingestor to automate the generation of embeddings and their export t

Superduper is an AI agent development kit and LLM application framework designed to build autonomous agents and data-driven applications. It functions as a RAG orchestration platform and vector search infrastructure, coordinating AI models with database storage to perform multi-step computations and actions using persisted data states. The project distinguishes itself by providing a database-integrated machine learning pipeline that executes training and inference tasks directly on data hosted within SQL and NoSQL databases. It allows for the deployment of self-hosted AI infrastructure on pri

zvec is an embedded vector database engine and indexing library designed for high-dimensional similarity search. It functions as a hybrid search engine and a retrieval-augmented generation knowledge base, allowing for the storage and retrieval of dense and sparse vectors. The system is distinguished by its hybrid retrieval pipeline, which fuses vector similarity, full-text keyword matching, and scalar metadata filtering into single query operations. It supports a plugin-based model integration system for registering custom embedding models and rerankers, as well as language bindings for nativ

Verba is a retrieval-augmented generation interface and chatbot that uses Weaviate to provide factual answers based on private datasets. It functions as a vector database knowledge base, combining a hybrid search engine with an orchestration interface to connect various large language model providers and embedding services. The system differentiates itself through a RAG pipeline manager for adjusting text chunking rules and retrieval settings, alongside a 3D vector space visualization tool for analyzing the spatial organization and clustering of high-dimensional embeddings. It employs a modul

SuperduperDB is an AI agent orchestrator and database-integrated machine learning platform. It serves as a framework for building stateful AI agents and retrieval-augmented generation applications by integrating large language models directly with database backends. The project enables the deployment of self-hosted AI infrastructure and the management of language models on private hardware using local checkpoints. It distinguishes itself by allowing users to attach AI components directly to data fields, triggering model execution and automated transformations based on database insertions and

Langroid is a multi-agent orchestration framework and tool integration suite designed for building complex AI applications. It serves as a multi-modal integration layer that connects diverse local and remote language models with an agentic retrieval-augmented generation system. The project distinguishes itself through a collaborative message-exchange paradigm, allowing specialized agents to delegate tasks hierarchically and coordinate via structured communication. It features an advanced state management system for conversational AI, including the ability to rewind and prune conversation hist

PostgresML is a machine learning database extension for PostgreSQL that integrates model training and inference directly into the database. It functions as an in-database AI platform and vector database, enabling the execution of large language models and natural language processing tasks on stored records without exporting data to external services. The system distinguishes itself by utilizing GPU acceleration to minimize latency during model predictions and employing a hybrid storage engine that maintains relational data alongside high-dimensional vectors. It allows for the building and fin

RedisInsight is a graphical user interface and management tool for browsing, analyzing, and administering Redis databases. It provides a visual environment for exploring key-value data structures, managing database instances, and performing data analysis across different operating systems and deployments. The tool distinguishes itself by providing dedicated visual managers for complex operations, including a vector database manager for configuring embeddings and similarity searches, a query workbench for executing raw commands and Lua scripts, and a performance monitoring dashboard for tracki

AdalFlow is an autonomous AI agent framework and LLM application library designed for building modular workflows. It serves as a model-agnostic interface and RAG pipeline orchestrator, allowing users to develop ReAct agents that utilize iterative reasoning and external tool execution to solve complex tasks. The project distinguishes itself through a prompt optimization system that uses textual gradient descent to automatically refine prompt templates and few-shot examples. It treats model feedback as a differentiable signal, enabling a form of LLM backpropagation to iteratively improve output

PocketFlow is a graph-based framework for designing and executing large language model operations and reasoning patterns. It serves as an orchestrator for building goal-oriented autonomous agents, multi-agent systems, and retrieval-augmented generation pipelines. The system is distinguished by its ability to coordinate autonomous AI agents that use shared memory and tools to solve complex goals, supported by a structured output engine that enforces schema-consistent responses. It utilizes graph-based workflow orchestration to manage sequences of model operations and supports supervisor-based

Kreuzberg is a document extraction engine that converts PDFs, Office files, images, and over 90 other formats into clean, structured text and metadata. It is built around a compiled Rust core that can be used as a native library, a command-line tool, a REST API server, or a WebAssembly module for browser-based processing. The system is designed to run entirely on self-hosted infrastructure, with no data leaving the user's environment. What distinguishes Kreuzberg is its breadth of integration surfaces and its pipeline architecture. It exposes extraction capabilities through native bindings fo

LiteLLM is a unified gateway and proxy server designed to centralize access to over one hundred language model providers. It provides a standardized API interface that abstracts vendor-specific schemas, allowing developers to interact with diverse models through a single, consistent format. By acting as a central traffic management layer, it enables organizations to route, secure, and govern model interactions across multiple deployments. The platform distinguishes itself through its policy-driven architecture, which uses configuration-based routing to manage traffic distribution, load balanc

Vector similarity search extension for PostgreSQL.

This project is an artificial intelligence application starter kit and cloud deployment framework. It provides a pre-configured foundation for building AI applications, featuring integrated authentication, orchestration, and vector database connectivity. The framework serves as an implementation template for retrieval augmented generation systems. It includes a pipeline to convert markdown documents into vector embeddings and store them in a database to enable question-and-answer functionality. The system covers the coordination of communication between user interfaces and backend AI service

OpenGpt is an agent orchestration platform and multimodal interface designed for building and deploying specialized AI personas. It allows users to create task-oriented agents with custom system prompts and behavioral constraints to automate professional, creative, and technical workflows. The project features a prompt engineering workflow that transforms simple user inputs into structured instructions to improve model accuracy. It integrates retrieval-augmented generation by connecting vector databases to the chat interface, enabling context-aware responses from private datasets. The platfo

Qdrant is a high-performance vector similarity database designed to store, index, and search high-dimensional vectors alongside structured metadata. It functions as a distributed search engine that manages large-scale data clusters, providing low-latency retrieval and complex filtering capabilities. The system is built to serve as a specialized middleware layer, connecting machine learning pipelines and AI agents to persistent storage for intelligent information retrieval and recommendation tasks. The platform distinguishes itself through advanced retrieval techniques, including support for h

This project is an artificial intelligence gateway that functions as a centralized middleware layer for managing, securing, and observing interactions with language, vision, and audio models. It provides a unified interface that standardizes requests across multiple providers, enabling teams to integrate AI capabilities into their applications through a consistent set of tools and protocols. The gateway distinguishes itself through its comprehensive infrastructure governance and traffic management capabilities. It allows for policy-driven routing, automated failover, and load balancing across

This project is a Model Context Protocol server and AI agent database connector. It provides a standardized communication layer that allows language models to interact with relational data stores, read database schemas, and manage PostgreSQL database resources. The implementation acts as a serverless host for the Model Context Protocol, deploying on distributed edge functions to connect AI assistants to a project. This enables AI agents to perform database administration, execute SQL queries, and handle schema migrations through an AI-compatible interface. The system covers broader capabilit

UltraRAG is an LLM RAG orchestration platform and AI agent research framework designed to coordinate complex retrieval-augmented generation workflows. It functions as a multimodal RAG engine capable of retrieving and generating responses using text, images, and diverse data types, while providing tools for vector database management and RAG performance evaluation. The platform features a visual RAG pipeline builder that uses a canvas interface to construct and debug data flows, synchronizing visual designs directly with underlying code. It distinguishes itself through an autonomous research s

LangChain4j is a framework and library for building applications powered by large language models on the JVM. It provides a unified API for developing AI agents, implementing retrieval augmented generation, and integrating generative AI capabilities into professional software built with frameworks like Spring Boot or Quarkus. The project enables the creation of autonomous agents that can reason through tasks, manage memory, and execute external tools to achieve specific goals. It differentiates itself through a unified model interface that allows developers to switch between multiple model pr

This project provides a dockerized AI workflow stack and orchestration templates for deploying a self-hosted AI environment. It establishes a localized infrastructure for building autonomous agents and model chains that process private data on-premises without external cloud dependencies. The environment is designed to support autonomous agent development, allowing models to dynamically select tools, execute shell commands, and interact with local file systems. It includes integrated vector database support to enable retrieval augmented generation and private document analysis. The stack cov

Rig is a framework for building large language model applications, featuring a multi-provider client and a workflow builder for retrieval-augmented generation systems. It serves as an orchestrator for creating autonomous agents that can maintain conversation state and execute complex tasks through custom prompting and plugins. The project provides standardized interfaces for both completion and embedding model providers, allowing for unified request and response patterns across different engines. It also includes a vector database integration layer that defines a common interface for indexing

FlagEmbedding is a comprehensive toolkit designed for training, benchmarking, and deploying embedding models, retrieval systems, and augmented generation pipelines. It provides the necessary infrastructure to transform text into high-dimensional vector representations and organize them into searchable structures for semantic search applications. The framework distinguishes itself through specialized capabilities for fine-tuning pre-trained embedding and reranking models on domain-specific datasets. By allowing users to adapt models to unique vocabularies and specialized retrieval tasks, it en

This project is a comprehensive framework for building AI-powered applications, providing a unified toolkit for orchestrating language models, autonomous agents, and interactive user interfaces. It serves as a central library for managing the entire lifecycle of AI interactions, from initial prompt generation and model provider abstraction to complex, multi-step reasoning and tool execution. The framework distinguishes itself through its deep integration with frontend development, specifically by enabling generative user interfaces that render dynamic components directly from model outputs. I