Awesome GitHub RepositoriesSchema Inference

q is a command-line utility for the processing, filtering, and aggregation of tabular text and database files using standard SQL syntax. It functions as a query engine that treats CSV and TSV files, as well as standard input, as relational database tables. The tool distinguishes itself by providing a persistent cache layer that stores processed tabular data in a binary format to accelerate repeated queries on large datasets. It also maps individual filenames or stream identifiers to relational table names, enabling SQL joins across disparate text files. The project covers a broad range of da

This is a cross-platform framework for building, training, and deploying custom machine learning models within the .NET ecosystem. It provides a predictive modeling engine for classification, regression, and forecasting tasks, alongside an inference runtime to generate predictions across different hardware architectures. The framework includes a gradient boosting library and supports interoperability with external models via a standardized open format. It features tools for prediction explainability, allowing the analysis of feature importance to debug model behavior and identify bias. The p

Hypothesis is a Python property-based testing library and data generation engine. It enables the discovery of edge cases and bugs by generating a wide range of randomized inputs based on defined strategies and shrinking complex failing examples to their smallest possible form. It also functions as a state machine testing framework to verify system behavior across sequences of interdependent operations. The project features a fuzzing integration layer that converts raw byte buffers from coverage-guided fuzzers into structured test cases. It includes a persistence mechanism to store and synchro

SandDance is a hardware-accelerated visualization library and web-based data explorer designed for the interactive analysis of large, non-aggregated datasets. It functions as an interactive data visualization tool that renders complex datasets and intricate visuals within a browser. The project provides an embeddable data canvas consisting of web components and tags, allowing for the integration of full visualization interfaces and interactive charts into external web applications. It utilizes WebGL hardware acceleration to efficiently render large volumes of data as interactive graphics. Th

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

Pinot is a distributed, columnar analytical database designed for high-concurrency, low-latency query processing. It functions as a real-time OLAP datastore, enabling interactive, user-facing analytics by ingesting and querying massive datasets from both streaming and batch sources. The system architecture relies on a centralized controller for cluster coordination and a distributed segment-based storage model to ensure horizontal scalability. The platform distinguishes itself through a hybrid ingestion pipeline that unifies real-time event streams and historical batch data into a single quer

Cocoindex is an incremental data processing engine that builds and maintains live indexes for AI agents, with a core focus on codebase indexing and knowledge graph extraction. The engine uses a function-graph execution model where user-defined Python functions are composed into a directed acyclic graph, and it processes data incrementally so only changed source records or code paths are re-computed, avoiding full recomputation at any scale. It supports automatic schema inference from transformation pipeline type annotations and provides full data lineage tracing, tagging every output record wi

GreptimeDB is a distributed, open-source time-series database built for unified observability. It stores and queries metrics, logs, and traces together in a single columnar engine, supporting both SQL and PromQL for analysis. The database is designed as a Kubernetes-native operator with a decoupled compute and storage architecture, enabling horizontal scaling and multi-region deployment. What distinguishes GreptimeDB is its role as a multi-protocol ingestion gateway, accepting data through OpenTelemetry, Prometheus Remote Write, InfluxDB, Loki, Elasticsearch, Kafka, and MQTT protocols without

jsonparser is a Go library for parsing, extracting, and modifying JSON data by key path without requiring predefined structs or type schemas. It uses a deterministic state machine to parse JSON directly from byte slices, enabling zero-copy traversal and lazy typed value extraction. The library provides key-path-based navigation to retrieve raw bytes and typed values from nested JSON structures, supporting extraction of strings, integers, floats, and booleans. It also enables in-place value modification and deletion at specified key paths, as well as iteration over arrays and objects using cal

dlt es una herramienta de ingesta de datos en Python y framework de pipeline ETL diseñado para obtener datos de diversas fuentes y persistirlos en destinos estructurados. Funciona como un motor de inferencia de esquemas que detecta automáticamente tipos de datos y aplana estructuras JSON anidadas en tablas relacionales, moviendo datos desde fuentes a lakehouses, almacenes de datos o bases de datos vectoriales. El proyecto destaca por la generación de pipelines impulsada por IA, utilizando modelos de lenguaje de gran tamaño para crear código de extracción y conectores para APIs REST. También admite almacenamiento vectorial multimodal y población especializada de bases de datos vectoriales para soportar aplicaciones de IA y machine learning. El framework cubre una amplia gama de capacidades, incluyendo evolución automática de esquemas, carga incremental de datos mediante seguimiento de estado y validación de calidad de datos mediante la aplicación de contratos de datos. Proporciona herramientas para la normalización de datos relacionales, transformaciones pre y post-carga, y una variedad de adaptadores de destino para bases de datos SQL y almacenes de objetos en la nube. La observabilidad se maneja a través de paneles de ejecución de pipelines, seguimiento de linaje de columnas y verificación de versiones de esquema mediante hashes basados en contenido.

swagger-php es un generador de especificaciones OpenAPI y biblioteca de documentación de API para PHP. Escanea atributos y anotaciones del código fuente de PHP para producir archivos de especificación estandarizados y referencias de API interactivas. La herramienta funciona como un motor de inferencia de esquemas que deriva tipos de datos y esquemas a partir de sugerencias de tipo nativas de PHP y bloques genéricos. Utiliza análisis estático basado en reflexión para extraer metadatos sin ejecutar la lógica de la aplicación y admite la herencia recursiva de esquemas para fusionar definiciones de clases e interfaces padre. El proyecto incluye una utilidad de línea de comandos para exportar definiciones de API a archivos YAML o JSON estáticos. También proporciona un pipeline para el ensamblaje de especificaciones con lógica de procesamiento conectable para modificar cómo se analiza o estructura el código fuente.

Daft is a distributed dataframe library and multimodal data processor designed to handle large-scale structured and unstructured data. It functions as a vectorized execution engine that processes tables alongside images, audio, and video, utilizing a unified schema to manage diverse data types. The project distinguishes itself by combining distributed data engineering with large-scale AI inference. It provides an AI data pipeline for batch-optimizing model prompts and generating high-dimensional text embeddings, while utilizing zero-copy memory sharing to execute custom Python functions witho

Pandera is a data pipeline validation framework and statistical type validation tool. It functions as a library for defining and enforcing schemas on datasets to ensure data quality and consistency, specifically providing validation capabilities for Pandas dataframes. The project includes a schema inference tool that automates setup by analyzing existing dataset samples to generate validation schemas. It also serves as a synthetic data generator, creating artificial datasets based on predefined schemas to verify data-producing functions. The framework covers data engineering quality assuranc

dsq is a command-line interface and data engine for executing SQL queries against local structured files, such as CSV, JSON, Parquet, and Excel, without requiring a formal database import. It functions as a schema-inference engine that automatically detects data types and maps heterogeneous file structures into relational tables for analysis. The tool utilizes a lazy stream data processor and checksum-based disk caching to handle large datasets with minimal memory usage. It provides a persistent interactive shell for iterative data exploration, allowing users to inspect inferred schemas and r

qsv is a high-performance command line toolkit for querying, transforming, and analyzing comma-separated value files. It functions as a data wrangling interface and a tabular data profiler, featuring a query engine capable of executing SQL statements and joins directly on flat files without requiring a database. The project is distinguished by its ability to process massive datasets that exceed available system memory. This is achieved through disk-based external memory processing, including multithreaded merge sorting, on-disk hash tables for deduplication, and lightweight file indexing for

Esta librería es un framework de procesamiento de datos para la JVM que proporciona un entorno con seguridad de tipos para manipular datos tabulares estructurados. Funciona como un conjunto de herramientas integral para realizar transformaciones de datos complejas, agregaciones y análisis estadístico, aprovechando la validación de esquemas en tiempo de compilación para garantizar la integridad estructural en los pipelines de datos. El proyecto se distingue por su profunda integración con entornos de notebooks interactivos y su uso de generación de código en tiempo de compilación. Al derivar y aplicar esquemas automáticamente a partir de entradas sin procesar, genera descriptores de acceso con seguridad de tipos que permiten el autocompletado en el IDE y la verificación estática de nombres de columnas. Esta arquitectura permite a los desarrolladores realizar procesamiento de pipelines funcional manteniendo una estricta seguridad de tipos, evitando eficazmente errores en tiempo de ejecución durante la manipulación de datos. La librería admite una amplia gama de flujos de trabajo de datos, incluyendo la importación y mapeo de esquemas de bases de datos relacionales, la realización de análisis geoespaciales y la ejecución de pivotes de datos complejos. Incluye amplias utilidades para la construcción, filtrado, ordenamiento y cálculo de estadísticas descriptivas de datos. Además, el framework proporciona capacidades robustas de visualización y generación de informes, permitiendo a los usuarios renderizar tablas HTML interactivas, componer documentos y generar gráficos directamente a partir de conjuntos de datos estructurados. La librería está diseñada para un uso fluido dentro de entornos de desarrollo Kotlin y Java, con soporte especializado para la gestión automatizada de dependencias e integración de kernels en notebooks interactivos.