ydataai/pandas-profiling
Pandas Profiling
This project is an exploratory data analysis framework and profiling tool designed to generate comprehensive statistical reports from Pandas and Spark DataFrames. It functions as a data quality profiler that identifies missing values, duplicates, and high correlations within tabular datasets.
The tool distinguishes itself through specialized capabilities for time-series analysis, extracting temporal statistics, seasonality, and auto-correlation plots. It also includes a dataset comparison utility to identify structural or content changes between different versions of a dataset.
The analysis surface covers automated data profiling, quality assessment, and unstructured content analysis. Results can be exported as standalone HTML files, JSON formats, or interactive notebook widgets.
A command-line interface is available to process CSV files and generate reports using configuration files.
Features
- Comprehensive Profiling Reports - Produces comprehensive exploratory data analysis reports detailing dataset statistics and quality.
- Automated Exploratory Analysis - Provides a framework that automatically generates statistical summaries and visual insights from tabular datasets.
- Data Profiling - Produces standardized HTML or JSON reports of dataset characteristics with minimal code.
- Data Quality - Detects missing values, duplicates, and high correlations to ensure data integrity before processing.
- Profiling Reports - Generates detailed exploratory data analysis reports and descriptive statistics for Pandas and Spark DataFrames.
- Data Quality Monitors - Identifies problematic data patterns including missing values, high correlation, and duplicates.
- Agnostic Interfaces - Implements a unified interface that allows the same analysis logic to run on both Pandas and Spark dataframes.
- Data Quality Profilers - Identifies missing values, duplicates, and high correlations within large tabular datasets.
- HTML Analysis Reports - Generates portable standalone HTML reports containing embedded visualizations and statistical summaries.
- Profiling Tools - A specialized profiling tool for extracting temporal statistical information, including auto-correlation, seasonality, and ACF/PACF plots.
- Statistical Analysis Libraries - Provides a comprehensive set of tools for calculating descriptive statistics and correlations across datasets.
- Tabular Data Type Inference - Automatically detects column data types in tabular data to select the most appropriate statistical analysis methods.
- Dataset Comparators - Generates comparative analyses between multiple versions of a dataset to identify structural or content changes.
- Time Series Analysis - Applies specialized temporal logic to identify seasonality and autocorrelation in time-indexed columns.
- Time Series Analysis Tools - Extracts temporal statistics, seasonality, and auto-correlation plots from time-dependent data.
- Data Exploration - Generates HTML profiling reports for dataframes.
