pandas-profiling/pandas-profiling
Pandas Profiling
This project is an exploratory data analysis library and profiling tool for Pandas and Spark DataFrames. It automates the initial investigation of datasets by generating comprehensive descriptive analysis reports, statistical summaries, and data quality warnings.
The system functions as a data quality profiler to detect missing values, duplicate rows, and type inconsistencies. It includes a dataset comparison tool for identifying structural and content shifts between different versions of the same data, as well as specialized tools for time-series analysis to calculate auto-correlation and seasonality.
The analytical surface covers automated text analysis for detecting scripts and character blocks, and metadata extraction for files and images. Results can be exported as HTML files, JSON structures, or interactive notebook widgets.
The library can be used through a Python API or via a command-line interface for processing CSV files using configuration files.
Features
- Profiling Reports - Provides comprehensive statistical summaries and data quality assessments generated directly from Pandas and Spark dataframes.
- Automated Exploratory Analysis - Automatically generates statistical summaries and visual insights to facilitate the initial investigation of datasets.
- Dataset Comparators - Provides a comparative reporting tool to identify structural and content shifts between different versions of a dataset.
- Dataframe Processing - Implements a structured pipeline that processes Pandas and Spark dataframes through sequential statistical and type-inference stages.
- Data Quality Profilers - Identifies missing values, duplicate rows, and type inconsistencies to ensure tabular dataset integrity.
- Time Series Analysis - Calculates time-dependent statistics including auto-correlation, seasonality, and partial auto-correlation plots.
- Modular Analyzers - Utilizes a modular architecture with independent pluggable analyzers to compute various statistical metrics and quality indicators.
- Tabular Data Type Inference - Automatically determines the semantic data types of columns by analyzing patterns and distributions within tabular data.
- Text Analysis Tools - Analyzes textual columns to identify common categories, character blocks, and scripts.
- JSON Serializers - Serializes all calculated profiling results into JSON format to decouple data analysis from the final visual report.
- Analysis Result Exporters - Exports profiling results into various formats including HTML files, JSON, and interactive notebook widgets.
- Profiling Executables - Includes a command-line interface for processing CSV files and generating reports without requiring custom Python code.
- Configuration-Driven Analysis - Provides an external configuration system to control the depth and specific calculators used during the profiling process.
- HTML Template Renderers - Generates comprehensive visual reports by injecting profiling metadata into predefined HTML and CSS templates.
- Statistics - Automated HTML profiling reports for dataframes.
- Python Visualization - Generates statistical analytic reports with integrated visualizations.
- Python Visualization Libraries - Generates statistical reports and visualizations for data analysis.
