Awesome GitHub RepositoriesData Quality Profilers

This library provides a diagnostic toolkit for automated data profiling and exploratory analysis. It generates comprehensive statistical summaries and visual reports for tabular datasets, enabling users to identify distribution patterns, missing values, and quality anomalies through a unified interface. The project distinguishes itself by offering differential analysis, which allows for the comparison of two dataset versions to track structural and statistical changes over time. It supports large-scale data processing through lazy evaluation and provides interactive widgets that embed directl

This project is an exploratory data analysis framework and profiling tool designed to generate comprehensive statistical reports from Pandas and Spark DataFrames. It functions as a data quality profiler that identifies missing values, duplicates, and high correlations within tabular datasets. The tool distinguishes itself through specialized capabilities for time-series analysis, extracting temporal statistics, seasonality, and auto-correlation plots. It also includes a dataset comparison utility to identify structural or content changes between different versions of a dataset. The analysis

This project is an exploratory data analysis library and profiling tool for Pandas and Spark DataFrames. It automates the initial investigation of datasets by generating comprehensive descriptive analysis reports, statistical summaries, and data quality warnings. The system functions as a data quality profiler to detect missing values, duplicate rows, and type inconsistencies. It includes a dataset comparison tool for identifying structural and content shifts between different versions of the same data, as well as specialized tools for time-series analysis to calculate auto-correlation and se

DataHub is a metadata management system and data catalog platform designed to provide a centralized directory for discovering, managing, and documenting datasets across a diverse data stack. It serves as a comprehensive framework for metadata management, incorporating a data governance framework to classify sensitive information and assign ownership for organizational accountability. The platform distinguishes itself through AI-enabled data discovery, which connects large language models to a metadata graph to allow for natural language search and exploration of data assets. It also provides

Ludwig is a declarative machine learning framework designed for training neural networks and large language models using configuration files instead of manual coding. It functions as a multimodal model builder and a low-code tool for supervised fine-tuning, allowing users to build models that process mixed inputs of text, images, audio, and tabular data. The project distinguishes itself through an automated hyperparameter optimizer and a system for large language model fine-tuning using parameter-efficient adapters. It features a multimodal data pipeline and the ability to automatically gener

This project is a plugin framework and agentic workflow library designed to connect large language models to professional toolstacks. It provides a system for integrating language models with external data warehouses, CRMs, and other enterprise software to retrieve and manipulate real-time business data. The framework enables the automation of specialized professional tasks through a file-based plugin definition system. It allows for the customization of domain expertise and plugin behavior to align with internal company processes, supported by an enterprise data connector that links models t

Feast is a machine learning feature store and MLOps data infrastructure layer. It provides a centralized system for managing and serving features across offline training and online production environments, utilizing an online feature serving layer for low-latency retrieval. The project centers on a feature registry that acts as a central catalog for defining, governing, and discovering feature services. It employs a unified data access layer to decouple feature retrieval from physical storage and includes a point-in-time data generator to create historically accurate training datasets that pr

Evidently is an AI observability platform and evaluation framework designed to quantify the performance of machine learning models and large language models. It functions as a monitoring tool for detecting data drift and quality degradation in tabular datasets, while providing a specialized analyzer for the faithfulness and correctness of retrieval augmented generation systems. The project distinguishes itself through an evaluation framework that utilizes judge models and custom rubrics to score language model outputs. It includes tools for iterative prompt optimization and the generation of

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

RedPajama-Data هي مجموعة أدوات لمعالجة مجموعات البيانات النصية واسعة النطاق المستخدمة لتدريب النماذج اللغوية الكبيرة. توفر خط أنابيب معالجة يركز على تنظيف، وإزالة التكرار، وتسجيل مجموعات ضخمة من النصوص لضمان جودة البيانات وتنوعها. يستخدم المشروع إطار عمل لتسجيل جودة المستندات يستخدم التعلم الآلي والاستدلالات الإحصائية لتقييم ما إذا كانت المستندات مناسبة للتدريب. يتضمن خط أنابيب تصفية مجموعات البيانات الذي يستخدم المصنفات والقوائم السوداء لإزالة الكلمات أو روابط URL غير المرغوب فيها. يتميز النظام بمجموعة أدوات لإزالة تكرار النصوص تقضي على المحتوى الزائد باستخدام تقنيات المطابقة الدقيقة والتقريبية. تسمح هذه القدرات بتحديد وإزالة المستندات المكررة أو المتطابقة تقريباً عبر مجموعة بيانات.

Amundsen is a data catalog and discovery platform that provides a centralized directory for indexing tables and dashboards. It functions as a metadata management system and search engine, allowing users to locate and understand available data assets across diverse distributed sources. The platform includes capabilities for data lineage tracking to map the origin and movement of datasets between systems. It also serves as a data profiling tool, calculating distribution and quality statistics for individual table columns to provide automated insights into the nature of the data. The system man

Visual Insights is an automated exploratory data analysis platform and causal inference tool designed to discover patterns and cause-and-effect relationships within datasets. It functions as an interactive data visualization library using a grammar-of-graphics approach to generate multi-dimensional charts and dashboards. The project distinguishes itself through a natural language interface that translates plain-text questions into data answers and visualizations via a language model. It provides a specialized framework for causal discovery and inference, allowing users to identify variable li

هذا المشروع عبارة عن مجموعة من المواد المرجعية والمبادئ التوجيهية لتنفيذ أطر تدقيق البيانات. يعمل كدليل مرجعي لجودة البيانات ودليل للتحقق من مجموعات البيانات لتحديد الأخطاء الهيكلية والإحصائية الشائعة في مجموعات البيانات. يوفر المشروع قاعدة معرفية منظمة لتنظيف البيانات، تتميز بكتالوج لأخطاء البيانات الواقعية واستراتيجيات عملية لاكتشافها وحلها. يتضمن أطراً محددة لتقييم مصدر البيانات وموثوقية المعلومات المجمعة. تغطي المادة مجموعة واسعة من قدرات تحليل البيانات، بما في ذلك التحقق من السلامة الإحصائية لاكتشاف التلاعب، وتقييمات صلاحية أخذ العينات لتحديد تحيز السكان، وطرق اكتشاف الأخطاء الهيكلية مثل مشاكل الترميز. كما يصف عمليات استعادة المعلومات الجدولية من المستندات المرئية من خلال التعرف الضوئي على الحروف (OCR).

qsv is a high-performance command line toolkit for querying, transforming, and analyzing comma-separated value files. It functions as a data wrangling interface and a tabular data profiler, featuring a query engine capable of executing SQL statements and joins directly on flat files without requiring a database. The project is distinguished by its ability to process massive datasets that exceed available system memory. This is achieved through disk-based external memory processing, including multithreaded merge sorting, on-disk hash tables for deduplication, and lightweight file indexing for