Pgai

pgai is a PostgreSQL AI toolkit and framework designed to integrate large language models and vector embeddings directly into a database. It serves as a bridge for executing machine learning model requests and performing text-to-SQL translations within standard database queries.

The project provides an automated vector embedding pipeline that handles the loading, parsing, and chunking of text from tables and unstructured documents. This system utilizes a background worker to synchronize embeddings automatically as source data changes and includes specialized tools for building retrieval-augmented generation applications and semantic search engines.

The toolkit covers broad capability areas including unstructured data processing with OCR, the creation of semantic catalogs to map database schemas to natural language, and the implementation of high-performance similarity searches through vector indexing and result reranking. It also enables data enrichment, classification, and content moderation by calling external models via SQL.

Features

AI Model Integrations - Integrates external AI models directly into PostgreSQL via SQL queries for natural language processing and embedding generation.
Database AI Toolkits - Provides a comprehensive set of tools to integrate LLMs and vector embeddings directly into PostgreSQL.
AI Model Integrations - Integrates external machine learning models directly into PostgreSQL queries for data enrichment and classification.
SQL-Based Machine Learning - Enables executing machine learning model requests and inference directly within standard SQL queries.
Standard RAG Development - Builds retrieval augmented generation pipelines that combine database retrieval with language models for grounded responses.
Retrieval-Augmented Generation - Implements the full retrieval-augmented generation pipeline by combining semantic search results with language model prompts.
SQL-Based Model Invocations - Allows executing model requests for text generation, classification, and moderation directly within SQL queries.
RAG Pipelines - Implements workflows that augment language model outputs by retrieving and integrating relevant external database data.
RAG Application Frameworks - Provides a framework to build RAG applications by combining retrieved database context with model prompts.
RAG Frameworks - Provides a framework for building retrieval-augmented generation applications using database context and LLM prompts.
SQL-Based Model Invocations - Executes requests to external machine learning models directly from within data queries.
Text-to-SQL Translators - Translates natural language user queries into executable SQL statements using semantic catalogs and schema descriptions.
Semantic Catalogs - Maintains semantic catalogs that map database objects to natural language descriptions to improve text-to-SQL accuracy.
Vector Embeddings - Generates numerical vector representations of database tables and files to enable semantic search.
Semantic Vector Search - Retrieves relevant data by calculating the mathematical distance between query and document embeddings.
Vector Similarity Search - Performs high-dimensional similarity searches to retrieve data based on semantic meaning rather than keywords.
Automatic Vector Embeddings - Automatically converts database content into vector representations using a background worker process.
Embedding Synchronization - Automatically synchronizes vector embeddings as source table data changes using state-based tracking.
Document Chunking and Embedding Pipelines - Provides automated pipelines that handle the full flow of chunking, embedding, and storing document data.
Embedding Pipelines - Implements modular pipelines that automate the loading, parsing, and formatting of data into vector embeddings.
Vector-Augmented Queries - Combines semantic vector search with model prompting within standard database queries to build knowledge-aware applications.
Semantic Search Engines - Implements a semantic search engine that retrieves information based on conceptual meaning using vector embeddings.
Continuous Sync Engines - Implements engines that automatically synchronize vector embeddings as the underlying source data changes.
In-Database Model Invocation - Enables executing external machine learning model requests and text-to-SQL translations directly within standard database queries.
Vector Indexing - Creates and manages indexes optimized for high-dimensional vector data to support semantic search.
Embedding Generation - Provides an automated pipeline to convert database content into high-dimensional vector representations using external workers.
Recursive Text Splitting - Implements recursive splitting functions to divide large bodies of text into chunks for AI model consumption.
Text Chunks - Splits long text into smaller segments using configurable algorithms and metadata injection for RAG context windows.
Result Reranking - Scores and reorders search results against a query to improve precision and relevance.
Embedding Status Monitors - Tracks the status of asynchronous embedding jobs, including pending items and processing states.
Data Processing - Uses large language models to enrich relational data through automated summarization, categorization, and content moderation.
Document Parsing and Extraction - Extracts text from unstructured PDFs and images using OCR to prepare content for vectorization and LLM ingestion.
Schema Description Generators - Generates natural language descriptions of database objects to help AI models understand technical schemas.
Embedding Synchronization Schedulers - Automates the periodic processing of updated data to synchronize embeddings via a background job system.
Document and Unstructured Extraction - Extracts text from PDFs and images using OCR and layout-aware parsing to prepare unstructured content for embedding.
Batch Embedding Management - Manages large-scale batch processing of embeddings with built-in resilience against failures and API rate limits.
Query Performance Tuning - Optimizes vector query performance by creating specialized indexes on embedding columns to reduce latency.
Semantic Knowledge Base Search - Retrieves relevant context and domain knowledge from the database using natural language queries and embeddings.
Background Job Processing - Provides a background worker system to process large-scale embedding tasks and synchronization jobs outside the main request flow.
Catalog Management - Enables the management of multiple independent semantic catalogs and embedding configurations within one environment.
Spatial and Vector Data - Simplifies creating and synchronizing vector embeddings.

casibase/casibase

4,443View on GitHub

Casibase is an open-source platform that orchestrates multi-turn conversations with large language models and manages retrieval-augmented knowledge bases from a single interface. It provides a unified system for connecting to over 30 AI model providers, ingesting documents into vector embeddings for semantic search, and running autonomous agent loops that can drive a browser, search the web, execute commands, and integrate with external tools. The platform distinguishes itself by combining AI conversation management with infrastructure and application orchestration capabilities. It includes a

chonkie-inc/chonkie

4,170View on GitHub

Chonkie is a text chunking library designed for retrieval-augmented generation pipelines. It functions as a semantic text splitter and RAG ingestion pipeline, transforming raw text into embedded segments for storage in vector databases. The project distinguishes itself through specialized splitting strategies, including an AST-based code splitter for preserving logical boundaries in source code and a semantic text splitter that uses embedding models to determine boundaries based on meaning. It also provides a vector database ingestor to automate the generation of embeddings and their export t

docker/genai-stack

5,333View on GitHub

This project is a containerized development stack and application framework for building retrieval-augmented generation systems. It provides a dockerized AI sandbox that integrates local model runtimes, knowledge graphs, and vector stores to enable the creation of contextual chatbots. The stack is distinguished by its graph-based vector store, which combines structured knowledge graphs with vector indices for both semantic and structural data retrieval. It allows for local model hosting with CPU or GPU acceleration, enabling generative tasks without reliance on external cloud APIs. The frame

datawhalechina/all-in-rag

3,989View on GitHub

This project is a retrieval augmented generation framework designed to build pipelines that connect unstructured data and knowledge graphs with large language models. It functions as a vector database orchestrator for indexing text and multimodal content, as well as a system for translating natural language queries into structured database commands. The framework integrates a hybrid retrieval engine that combines dense vector search with sparse keyword matching to increase the precision of retrieved contexts. It further enhances reasoning and relationship mapping through a graph-augmented ret

timescalepgaiArchived

Features