Awesome GitHub RepositoriesData Quality Profilers

This library provides a diagnostic toolkit for automated data profiling and exploratory analysis. It generates comprehensive statistical summaries and visual reports for tabular datasets, enabling users to identify distribution patterns, missing values, and quality anomalies through a unified interface. The project distinguishes itself by offering differential analysis, which allows for the comparison of two dataset versions to track structural and statistical changes over time. It supports large-scale data processing through lazy evaluation and provides interactive widgets that embed directl

This project is an exploratory data analysis framework and profiling tool designed to generate comprehensive statistical reports from Pandas and Spark DataFrames. It functions as a data quality profiler that identifies missing values, duplicates, and high correlations within tabular datasets. The tool distinguishes itself through specialized capabilities for time-series analysis, extracting temporal statistics, seasonality, and auto-correlation plots. It also includes a dataset comparison utility to identify structural or content changes between different versions of a dataset. The analysis

This project is an exploratory data analysis library and profiling tool for Pandas and Spark DataFrames. It automates the initial investigation of datasets by generating comprehensive descriptive analysis reports, statistical summaries, and data quality warnings. The system functions as a data quality profiler to detect missing values, duplicate rows, and type inconsistencies. It includes a dataset comparison tool for identifying structural and content shifts between different versions of the same data, as well as specialized tools for time-series analysis to calculate auto-correlation and se

DataHub is a metadata management system and data catalog platform designed to provide a centralized directory for discovering, managing, and documenting datasets across a diverse data stack. It serves as a comprehensive framework for metadata management, incorporating a data governance framework to classify sensitive information and assign ownership for organizational accountability. The platform distinguishes itself through AI-enabled data discovery, which connects large language models to a metadata graph to allow for natural language search and exploration of data assets. It also provides

Ludwig is a declarative machine learning framework designed for training neural networks and large language models using configuration files instead of manual coding. It functions as a multimodal model builder and a low-code tool for supervised fine-tuning, allowing users to build models that process mixed inputs of text, images, audio, and tabular data. The project distinguishes itself through an automated hyperparameter optimizer and a system for large language model fine-tuning using parameter-efficient adapters. It features a multimodal data pipeline and the ability to automatically gener

This project is a plugin framework and agentic workflow library designed to connect large language models to professional toolstacks. It provides a system for integrating language models with external data warehouses, CRMs, and other enterprise software to retrieve and manipulate real-time business data. The framework enables the automation of specialized professional tasks through a file-based plugin definition system. It allows for the customization of domain expertise and plugin behavior to align with internal company processes, supported by an enterprise data connector that links models t

Feast is a machine learning feature store and MLOps data infrastructure layer. It provides a centralized system for managing and serving features across offline training and online production environments, utilizing an online feature serving layer for low-latency retrieval. The project centers on a feature registry that acts as a central catalog for defining, governing, and discovering feature services. It employs a unified data access layer to decouple feature retrieval from physical storage and includes a point-in-time data generator to create historically accurate training datasets that pr

Evidently is an AI observability platform and evaluation framework designed to quantify the performance of machine learning models and large language models. It functions as a monitoring tool for detecting data drift and quality degradation in tabular datasets, while providing a specialized analyzer for the faithfulness and correctness of retrieval augmented generation systems. The project distinguishes itself through an evaluation framework that utilizes judge models and custom rubrics to score language model outputs. It includes tools for iterative prompt optimization and the generation of

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

RedPajama-Data ist ein Toolset für das Preprocessing großskaliger Textdatensätze, die zum Training großer Sprachmodelle verwendet werden. Es bietet eine Preprocessing-Pipeline, die sich auf das Bereinigen, Deduplizieren und Bewerten massiver Textsammlungen konzentriert, um Datenqualität und -vielfalt sicherzustellen. Das Projekt nutzt ein Framework zur Bewertung der Dokumentqualität, das Machine Learning und statistische Heuristiken einsetzt, um zu bewerten, ob Dokumente für das Training geeignet sind. Es enthält eine Datensatz-Filter-Pipeline, die Klassifikatoren und Blocklisten verwendet, um unerwünschte Wörter oder URLs zu entfernen. Das System verfügt über ein Text-Deduplizierungstoolset, das redundante Inhalte sowohl mit exakten als auch mit Fuzzy-Matching-Techniken eliminiert. Diese Funktionen ermöglichen die Identifizierung und Entfernung doppelter oder nahezu identischer Dokumente innerhalb eines Korpus.

Amundsen is a data catalog and discovery platform that provides a centralized directory for indexing tables and dashboards. It functions as a metadata management system and search engine, allowing users to locate and understand available data assets across diverse distributed sources. The platform includes capabilities for data lineage tracking to map the origin and movement of datasets between systems. It also serves as a data profiling tool, calculating distribution and quality statistics for individual table columns to provide automated insights into the nature of the data. The system man

Visual Insights ist eine Plattform für automatisierte explorative Datenanalyse und ein Tool für kausale Inferenz, das entwickelt wurde, um Muster sowie Ursache-Wirkungs-Zusammenhänge in Datensätzen zu entdecken. Es fungiert als interaktive Datenvisualisierungsbibliothek, die einen Grammar-of-Graphics-Ansatz verwendet, um mehrdimensionale Diagramme und Dashboards zu generieren. Das Projekt zeichnet sich durch eine natürlichsprachliche Schnittstelle aus, die Fragen in Klartext mithilfe eines Sprachmodells in Datenantworten und Visualisierungen übersetzt. Es bietet ein spezialisiertes Framework für kausale Entdeckung und Inferenz, das es Benutzern ermöglicht, Variablenverknüpfungen durch interaktive Kausaldiagramme zu identifizieren und What-if-Analysen zur Validierung von Hypothesen durchzuführen. Die Plattform deckt ein breites Spektrum an Funktionen ab, darunter visuelle Datenbereinigung, statistische Profilerstellung und automatisierte Datensatztransformation. Sie unterstützt die Integration verschiedener Daten aus lokalen Dateien und Remote-Datenbanken und verfügt über eine leistungsstarke Verarbeitungs-Engine für die lokale Handhabung großer Datensätze. Zusätzlich ermöglicht das System die Einbettung interaktiver Analysekomponenten in Webanwendungen und Notebooks.

Dieses Projekt ist eine Sammlung von Referenzmaterialien und Richtlinien für die Implementierung von Data-Audit-Frameworks. Es dient als Referenzleitfaden für Datenqualität und als Handbuch zur Datensatzvalidierung für die Identifizierung häufiger struktureller und statistischer Fehler in Datensätzen. Das Projekt bietet eine strukturierte Wissensbasis für Datenbereinigung, inklusive eines Katalogs realer Datenfehler und praktischer Strategien für deren Erkennung und Behebung. Es enthält spezifische Frameworks zur Evaluierung der Datenherkunft (Provenance) und der Zuverlässigkeit aggregierter Informationen. Das Material deckt ein breites Spektrum an Datenanalyse-Funktionen ab, einschließlich statistischer Integritätsvalidierung zur Erkennung von Manipulationen, Assessments der Stichprobengültigkeit zur Identifizierung von Populations-Bias und Methoden zur strukturellen Fehlererkennung wie Kodierungsprobleme. Zudem beschreibt es Prozesse zur Wiederherstellung tabellarischer Informationen aus visuellen Dokumenten mittels OCR (Optical Character Recognition).

qsv is a high-performance command line toolkit for querying, transforming, and analyzing comma-separated value files. It functions as a data wrangling interface and a tabular data profiler, featuring a query engine capable of executing SQL statements and joins directly on flat files without requiring a database. The project is distinguished by its ability to process massive datasets that exceed available system memory. This is achieved through disk-based external memory processing, including multithreaded merge sorting, on-disk hash tables for deduplication, and lightweight file indexing for