Awesome GitHub RepositoriesData Quality Profilers

This library provides a diagnostic toolkit for automated data profiling and exploratory analysis. It generates comprehensive statistical summaries and visual reports for tabular datasets, enabling users to identify distribution patterns, missing values, and quality anomalies through a unified interface. The project distinguishes itself by offering differential analysis, which allows for the comparison of two dataset versions to track structural and statistical changes over time. It supports large-scale data processing through lazy evaluation and provides interactive widgets that embed directl

This project is an exploratory data analysis framework and profiling tool designed to generate comprehensive statistical reports from Pandas and Spark DataFrames. It functions as a data quality profiler that identifies missing values, duplicates, and high correlations within tabular datasets. The tool distinguishes itself through specialized capabilities for time-series analysis, extracting temporal statistics, seasonality, and auto-correlation plots. It also includes a dataset comparison utility to identify structural or content changes between different versions of a dataset. The analysis

This project is an exploratory data analysis library and profiling tool for Pandas and Spark DataFrames. It automates the initial investigation of datasets by generating comprehensive descriptive analysis reports, statistical summaries, and data quality warnings. The system functions as a data quality profiler to detect missing values, duplicate rows, and type inconsistencies. It includes a dataset comparison tool for identifying structural and content shifts between different versions of the same data, as well as specialized tools for time-series analysis to calculate auto-correlation and se

DataHub is a metadata management system and data catalog platform designed to provide a centralized directory for discovering, managing, and documenting datasets across a diverse data stack. It serves as a comprehensive framework for metadata management, incorporating a data governance framework to classify sensitive information and assign ownership for organizational accountability. The platform distinguishes itself through AI-enabled data discovery, which connects large language models to a metadata graph to allow for natural language search and exploration of data assets. It also provides

Ludwig is a declarative machine learning framework designed for training neural networks and large language models using configuration files instead of manual coding. It functions as a multimodal model builder and a low-code tool for supervised fine-tuning, allowing users to build models that process mixed inputs of text, images, audio, and tabular data. The project distinguishes itself through an automated hyperparameter optimizer and a system for large language model fine-tuning using parameter-efficient adapters. It features a multimodal data pipeline and the ability to automatically gener

This project is a plugin framework and agentic workflow library designed to connect large language models to professional toolstacks. It provides a system for integrating language models with external data warehouses, CRMs, and other enterprise software to retrieve and manipulate real-time business data. The framework enables the automation of specialized professional tasks through a file-based plugin definition system. It allows for the customization of domain expertise and plugin behavior to align with internal company processes, supported by an enterprise data connector that links models t

Feast is a machine learning feature store and MLOps data infrastructure layer. It provides a centralized system for managing and serving features across offline training and online production environments, utilizing an online feature serving layer for low-latency retrieval. The project centers on a feature registry that acts as a central catalog for defining, governing, and discovering feature services. It employs a unified data access layer to decouple feature retrieval from physical storage and includes a point-in-time data generator to create historically accurate training datasets that pr

Evidently is an AI observability platform and evaluation framework designed to quantify the performance of machine learning models and large language models. It functions as a monitoring tool for detecting data drift and quality degradation in tabular datasets, while providing a specialized analyzer for the faithfulness and correctness of retrieval augmented generation systems. The project distinguishes itself through an evaluation framework that utilizes judge models and custom rubrics to score language model outputs. It includes tools for iterative prompt optimization and the generation of

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

RedPajama-Data es un conjunto de herramientas para el preprocesamiento de conjuntos de datos de texto a gran escala utilizados para entrenar modelos de lenguaje grandes. Proporciona una canalización de preprocesamiento centrada en la limpieza, deduplicación y puntuación de colecciones masivas de texto para garantizar la calidad y diversidad de los datos. El proyecto utiliza un framework de puntuación de calidad de documentos que emplea aprendizaje automático y heurísticas estadísticas para evaluar si los documentos son adecuados para el entrenamiento. Incluye una canalización de filtrado de conjuntos de datos que utiliza clasificadores y listas de bloqueo para eliminar palabras o URLs no deseadas. El sistema cuenta con un conjunto de herramientas de deduplicación de texto que elimina contenido redundante utilizando técnicas de coincidencia exacta y difusa. Estas capacidades permiten la identificación y eliminación de documentos duplicados o casi idénticos en un corpus.

Amundsen is a data catalog and discovery platform that provides a centralized directory for indexing tables and dashboards. It functions as a metadata management system and search engine, allowing users to locate and understand available data assets across diverse distributed sources. The platform includes capabilities for data lineage tracking to map the origin and movement of datasets between systems. It also serves as a data profiling tool, calculating distribution and quality statistics for individual table columns to provide automated insights into the nature of the data. The system man

Visual Insights es una plataforma de análisis exploratorio de datos automatizado y herramienta de inferencia causal diseñada para descubrir patrones y relaciones de causa y efecto dentro de los datasets. Funciona como una librería de visualización de datos interactiva utilizando un enfoque de gramática de gráficos para generar gráficos y dashboards multidimensionales. El proyecto se distingue por una interfaz de lenguaje natural que traduce preguntas en texto plano a respuestas y visualizaciones de datos mediante un modelo de lenguaje. Proporciona un framework especializado para el descubrimiento e inferencia causal, permitiendo a los usuarios identificar enlaces entre variables mediante gráficos causales interactivos y realizar análisis de tipo "qué pasaría si" (what-if) para validar hipótesis. La plataforma cubre un amplio rango de capacidades, incluyendo limpieza visual de datos, perfilado estadístico y transformación automatizada de datasets. Soporta la integración de datos diversos desde archivos locales y bases de datos remotas, y cuenta con un motor de procesamiento de alto rendimiento para manejar grandes datasets localmente. Además, el sistema permite embeber componentes de análisis interactivos en aplicaciones web y notebooks.

Este proyecto es una colección de materiales de referencia y directrices para implementar frameworks de auditoría de datos. Sirve como una guía de referencia de calidad de datos y un manual de validación de conjuntos de datos para identificar errores estructurales y estadísticos comunes en datasets. El proyecto proporciona una base de conocimiento estructurada para la limpieza de datos, presentando un catálogo de errores de datos del mundo real y estrategias prácticas para su detección y resolución. Incluye frameworks específicos para evaluar la procedencia de los datos y la fiabilidad de la información agregada. El material cubre una amplia gama de capacidades de análisis de datos, incluyendo validación de integridad estadística para detectar manipulación, evaluaciones de validez de muestreo para identificar sesgos de población y métodos para la detección de errores estructurales como problemas de codificación. También describe procesos para recuperar información tabular de documentos visuales mediante reconocimiento óptico de caracteres (OCR).

qsv is a high-performance command line toolkit for querying, transforming, and analyzing comma-separated value files. It functions as a data wrangling interface and a tabular data profiler, featuring a query engine capable of executing SQL statements and joins directly on flat files without requiring a database. The project is distinguished by its ability to process massive datasets that exceed available system memory. This is achieved through disk-based external memory processing, including multithreaded merge sorting, on-disk hash tables for deduplication, and lightweight file indexing for