timescale/pgaiArchived
Pgai
pgai ist ein PostgreSQL-KI-Toolkit und -Framework, das darauf ausgelegt ist, Large Language Models und Vektor-Embeddings direkt in eine Datenbank zu integrieren. Es dient als Brücke für die Ausführung von Anfragen an Machine-Learning-Modelle und die Durchführung von Text-zu-SQL-Übersetzungen innerhalb von Standard-Datenbankabfragen.
Das Projekt bietet eine automatisierte Pipeline für Vektor-Embeddings, die das Laden, Parsen und Chunking von Text aus Tabellen und unstrukturierten Dokumenten übernimmt. Dieses System nutzt einen Hintergrund-Worker, um Embeddings automatisch zu synchronisieren, wenn sich Quelldaten ändern, und enthält spezialisierte Tools für den Aufbau von RAG-Anwendungen (Retrieval-Augmented Generation) und semantischen Suchmaschinen.
Das Toolkit deckt breite Funktionsbereiche ab, darunter die Verarbeitung unstrukturierter Daten mittels OCR, die Erstellung semantischer Kataloge zur Abbildung von Datenbankschemata auf natürliche Sprache sowie die Implementierung von Hochleistungs-Ähnlichkeitssuchen durch Vektorindizierung und Result-Reranking. Zudem ermöglicht es Datenanreicherung, Klassifizierung und Content-Moderation durch den Aufruf externer Modelle via SQL.
Features
- AI Model Integrations - Integrates external AI models directly into PostgreSQL via SQL queries for natural language processing and embedding generation.
- Database AI Toolkits - Provides a comprehensive set of tools to integrate LLMs and vector embeddings directly into PostgreSQL.
- AI Model Integrations - Integrates external machine learning models directly into PostgreSQL queries for data enrichment and classification.
- SQL-Based Machine Learning - Enables executing machine learning model requests and inference directly within standard SQL queries.
- Standard RAG Development - Builds retrieval augmented generation pipelines that combine database retrieval with language models for grounded responses.
- Retrieval-Augmented Generation - Implements the full retrieval-augmented generation pipeline by combining semantic search results with language model prompts.
- SQL-Based Model Invocations - Allows executing model requests for text generation, classification, and moderation directly within SQL queries.
- RAG Pipelines - Implements workflows that augment language model outputs by retrieving and integrating relevant external database data.
- RAG Application Frameworks - Provides a framework to build RAG applications by combining retrieved database context with model prompts.
- RAG Frameworks - Provides a framework for building retrieval-augmented generation applications using database context and LLM prompts.
- SQL-Based Model Invocations - Executes requests to external machine learning models directly from within data queries.
- Text-to-SQL Translators - Translates natural language user queries into executable SQL statements using semantic catalogs and schema descriptions.
- Semantic Catalogs - Maintains semantic catalogs that map database objects to natural language descriptions to improve text-to-SQL accuracy.
- Vector Embeddings - Generates numerical vector representations of database tables and files to enable semantic search.
- Semantic Vector Search - Retrieves relevant data by calculating the mathematical distance between query and document embeddings.
- Vector Similarity Search - Performs high-dimensional similarity searches to retrieve data based on semantic meaning rather than keywords.
- Automatic Vector Embeddings - Automatically converts database content into vector representations using a background worker process.
- Embedding Synchronization - Automatically synchronizes vector embeddings as source table data changes using state-based tracking.
- Document Chunking and Embedding Pipelines - Provides automated pipelines that handle the full flow of chunking, embedding, and storing document data.
- Embedding Pipelines - Implements modular pipelines that automate the loading, parsing, and formatting of data into vector embeddings.
- Vector-Augmented Queries - Combines semantic vector search with model prompting within standard database queries to build knowledge-aware applications.
- Semantic Search Engines - Implements a semantic search engine that retrieves information based on conceptual meaning using vector embeddings.
- Continuous Sync Engines - Implements engines that automatically synchronize vector embeddings as the underlying source data changes.
- In-Database Model Invocation - Enables executing external machine learning model requests and text-to-SQL translations directly within standard database queries.
- Vector Indexing - Creates and manages indexes optimized for high-dimensional vector data to support semantic search.
- Embedding Generation - Provides an automated pipeline to convert database content into high-dimensional vector representations using external workers.
- Recursive Text Splitting - Implements recursive splitting functions to divide large bodies of text into chunks for AI model consumption.
- Text Chunks - Splits long text into smaller segments using configurable algorithms and metadata injection for RAG context windows.
- Result Reranking - Scores and reorders search results against a query to improve precision and relevance.
- Embedding Status Monitors - Tracks the status of asynchronous embedding jobs, including pending items and processing states.
- Data Processing - Uses large language models to enrich relational data through automated summarization, categorization, and content moderation.
- Document Parsing and Extraction - Extracts text from unstructured PDFs and images using OCR to prepare content for vectorization and LLM ingestion.
- Schema Description Generators - Generates natural language descriptions of database objects to help AI models understand technical schemas.
- Embedding Synchronization Schedulers - Automates the periodic processing of updated data to synchronize embeddings via a background job system.
- Document and Unstructured Extraction - Extracts text from PDFs and images using OCR and layout-aware parsing to prepare unstructured content for embedding.
- Batch Embedding Management - Manages large-scale batch processing of embeddings with built-in resilience against failures and API rate limits.
- Query Performance Tuning - Optimizes vector query performance by creating specialized indexes on embedding columns to reduce latency.
- Semantic Knowledge Base Search - Retrieves relevant context and domain knowledge from the database using natural language queries and embeddings.
- Background Job Processing - Provides a background worker system to process large-scale embedding tasks and synchronization jobs outside the main request flow.
- Catalog Management - Enables the management of multiple independent semantic catalogs and embedding configurations within one environment.
- Spatial and Vector Data - Simplifies creating and synchronizing vector embeddings.
