Data-Centric-AI-Community/fg-data-profiling
Fg Data Profiling
This project is a data profiling and exploratory data analysis tool designed to generate automated quality reports for Pandas and Spark dataframes. It serves as a system for computing descriptive statistics, identifying correlations, and analyzing univariate and multivariate data patterns.
The tool provides specialized capabilities for comparing different versions of datasets to identify changes in data quality and distributions. It includes a dedicated profiler for time-dependent data to extract statistical information such as seasonality and auto-correlation.
The software covers a broad analysis surface including text data composition, multivariate interaction mapping, and the extraction of technical file metadata. Analysis results can be exported as HTML files, JSON data, or interactive widgets.
A command line interface is available for processing CSV files and generating reports using local configuration files.
Features
- Data Profiling - Provides automated statistical profiling to analyze data patterns and validate data quality.
- Data Observability Profilings - Provides automated profiling of data distributions to detect shifts and quality issues in Pandas DataFrames.
- Univariate Statistical Profiling - Computes descriptive statistics and generates distribution histograms for individual variables.
- Automated - Provides a unified system for automated data quality reporting and profiling for both Pandas and Spark.
- Data Quality Reports - Produces human-readable exploratory data analysis reports featuring descriptive statistics and quality alerts.
- Statistical Aggregators - Computes descriptive statistics and distribution histograms for individual columns to determine central tendency.
- Automated Exploratory Analysis - Automatically generates statistical summaries and visual insights to discover patterns and anomalies in new datasets.
- Dataset Comparators - Provides utilities for calculating statistical differences and drift between different versions of dataframes.
- Agnostic Interfaces - Implements a unified API to execute profiling logic across both Pandas and Spark data structures.
- Distributed Dataframe Analysis - Implements large-scale data quality checks and exploratory analysis specifically for Spark DataFrames.
- Multivariate Relationship Visualizers - Calculates pairwise interactions and correlations between multiple variables to identify dataset dependencies.
- Time Series Analysis - Calculates temporal trends, seasonality, and auto-correlation for time-dependent datasets.
- Time-Series Statistical Profiling - Extracts statistical information from time-dependent data, including seasonality and auto-correlation plots.
- Time Series Decomposition - Analyzes temporal data using auto-correlation and partial auto-correlation to identify seasonality and trends.
- Analysis Result Exporters - Serializes profiling reports into HTML, JSON, and interactive widgets for external system integration.
- Time Series Analysis Tools - Provides specialized tools for calculating statistical properties of temporal data, such as seasonality and auto-correlation.
- Command Line Interfaces - Provides a command line tool for processing CSV files and generating profiling reports via local configurations.
- Analysis Report Serialization - Transforms analysis results into HTML, JSON, and interactive widgets for external viewing and integration.
- Data Analysis - Generates data profiling reports for DataFrames.
- Data Analysis and Processing - Data quality profiling for Pandas and Spark.
