Data-Centric-AI-Community/ydata-profiling
Ydata Profiling
This library provides a diagnostic toolkit for automated data profiling and exploratory analysis. It generates comprehensive statistical summaries and visual reports for tabular datasets, enabling users to identify distribution patterns, missing values, and quality anomalies through a unified interface.
The project distinguishes itself by offering differential analysis, which allows for the comparison of two dataset versions to track structural and statistical changes over time. It supports large-scale data processing through lazy evaluation and provides interactive widgets that embed directly into notebook environments for immediate inspection.
The tool covers a broad range of analytical capabilities, including automated reporting, data quality assessment, and version tracking. It supports multiple output formats, allowing findings to be exported as portable documents or integrated into automated data pipelines. The library is available as a Python package and provides consistent diagnostic utilities across different data processing frameworks.
Features
- Data Quality Profilers - Produces comprehensive statistical summaries and visual charts to detect quality problems and understand data distributions.
- Automated Exploratory Analysis - Automates the statistical summary and visualization of tabular datasets to identify patterns and quality issues.
- Spark Data Quality Analyzers - Summarizes large-scale datasets to detect inconsistencies and track changes across data pipelines.
- Automated Analytical Reports - Generates portable and comprehensive statistical summaries of datasets for stakeholders and automated pipelines.
- Pandas Profiling Tools - Generates automated statistical reports and visual summaries for tabular data to identify quality issues.
- Differential Dataset Comparisons - Provides differential analysis to track statistical and structural changes between two dataset versions.
- Dataset Versioning Platforms - Identifies structural and statistical differences between two versions of a dataset to track preprocessing impacts.
- Lazy Evaluation Frameworks - Defers heavy computations until report generation to optimize memory usage during large-scale data processing.
- Vectorized Statistical Aggregators - Calculates descriptive metrics across columns using vectorized operations to identify data distributions and quality anomalies.
- Dataset Comparison Reports - Highlights inconsistencies and changes between two data versions by generating a unified report.
- Notebook Analysis Widgets - Displays interactive profiling widgets directly inside data science environments for immediate exploration.
- Analytical Widget Serializers - Encodes analytical results into self-contained web assets for interactive inspection in notebook environments.
- Data Analysis and Processing - Data quality profiling for dataframes.
