Quartz/bad-data-guide
Bad Data Guide
This project is a collection of reference materials and guidelines for implementing data audit frameworks. It serves as a data quality reference guide and a dataset validation manual for identifying common structural and statistical errors in datasets.
The project provides a structured knowledge base for data cleaning, featuring a catalog of real-world data errors and practical strategies for their detection and resolution. It includes specific frameworks for evaluating data provenance and the reliability of aggregated information.
The material covers a broad range of data analysis capabilities, including statistical integrity validation to detect manipulation, sampling validity assessments to identify population bias, and methods for structural error detection such as encoding issues. It also describes processes for recovering tabular information from visual documents through optical character recognition.
Features
- Data Audit Frameworks - Implements a comprehensive framework for evaluating sampling validity, provenance, and aggregation reliability in datasets.
- Dataset Quality Auditing - Identifies common errors and unreliable patterns in real-world datasets to prevent mistakes during the analytical process.
- Data Auditing - Identifies patterns of unreliable data such as suspicious default values and inconsistent spelling.
- Data Cleaning Procedures - Provides workflows for finding and resolving structural errors and inconsistent values using expert-led mitigation strategies.
- Data Error Mitigation Strategies - Provides suggested solutions and mitigation strategies for data errors, categorized by the expert best equipped to fix them.
- Data Quality Profilers - References common real-world data errors and applies methods to resolve or mitigate those issues.
- Statistical Integrity Validators - Provides a framework for identifying analytical errors and deceptive patterns like p-hacking to ensure statistical integrity.
- Data Quality Guides - Provides a comprehensive catalog of real-world data errors with practical strategies for detection and resolution.
- Data Quality Taxonomies - Organizes data quality issues into a structured hierarchy based on their root cause and impact.
- Statistical Anomaly Heuristics - Implements rule-based heuristic checks to identify statistical anomalies and population sampling biases.
- Integrity Signature Matching - Identifies unreliable data by matching observed values against known signatures of software errors and human bias.
- Data Extraction and OCR - Extracts and recovers tabular information from PDFs and scanned images using OCR tools.
- OCR Document Parsers - Utilizes OCR-driven document parsing to extract structured tabular information from images and scanned PDFs.
- Document Data Extraction - Recovers structured tabular data from PDFs and scanned images using optical character recognition processes.
- Data Provenance Frameworks - Enables investigation of collection processes and tracking of manual changes to determine information reliability.
- Provenance Chain Analysis - Evaluates data reliability by tracing the chain of custody and manual modifications from source to final set.
- Structural Dataset Error Detection - Identifies technical issues such as truncated spreadsheets and garbled text encoding caused by software limits.
- Dataset Validation Manuals - Provides a technical reference for spotting glitches like encoding errors and truncated files in raw data.
- Benford's Law Analysis - Identifies deceptive data patterns using red flags like p-hacking and Benford's Law failures.
- Population Bias Evaluations - Evaluates whether a dataset is representative by checking for non-random samples and biased population collection.
- Sampling Representativeness Assessments - Checks for non-random samples and biased populations to determine if a dataset is representative.
- Machine Learning - Reference for identifying and handling problematic datasets.
