Quartz/bad-data-guide
Bad Data Guide
这是一个参考资料和指南集合，专注于实施数据审计框架。它作为一个数据质量参考指南和数据集验证手册，用于识别数据集中的常见结构性和统计性错误。
该项目提供了一个结构化的数据清洗知识库，包含真实世界数据错误的目录以及用于检测和解决这些错误的实用策略。它包括用于评估数据来源和聚合信息可靠性的特定框架。
该材料涵盖了广泛的数据分析功能，包括用于检测篡改的统计完整性验证、用于识别总体偏差的抽样有效性评估，以及诸如编码问题等结构性错误检测方法。它还描述了通过光学字符识别 (OCR) 从视觉文档中恢复表格信息的过程。
Features
- Data Audit Frameworks - Implements a comprehensive framework for evaluating sampling validity, provenance, and aggregation reliability in datasets.
- Dataset Quality Auditing - Identifies common errors and unreliable patterns in real-world datasets to prevent mistakes during the analytical process.
- Data Auditing - Identifies patterns of unreliable data such as suspicious default values and inconsistent spelling.
- Data Cleaning Procedures - Provides workflows for finding and resolving structural errors and inconsistent values using expert-led mitigation strategies.
- Data Error Mitigation Strategies - Provides suggested solutions and mitigation strategies for data errors, categorized by the expert best equipped to fix them.
- Data Quality Profilers - References common real-world data errors and applies methods to resolve or mitigate those issues.
- Statistical Integrity Validators - Provides a framework for identifying analytical errors and deceptive patterns like p-hacking to ensure statistical integrity.
- Data Quality Guides - Provides a comprehensive catalog of real-world data errors with practical strategies for detection and resolution.
- Data Quality Taxonomies - Organizes data quality issues into a structured hierarchy based on their root cause and impact.
- Statistical Anomaly Heuristics - Implements rule-based heuristic checks to identify statistical anomalies and population sampling biases.
- Integrity Signature Matching - Identifies unreliable data by matching observed values against known signatures of software errors and human bias.
- Data Extraction and OCR - Extracts and recovers tabular information from PDFs and scanned images using OCR tools.
- OCR Document Parsers - Utilizes OCR-driven document parsing to extract structured tabular information from images and scanned PDFs.
- Document Data Extraction - Recovers structured tabular data from PDFs and scanned images using optical character recognition processes.
- Data Provenance Frameworks - Enables investigation of collection processes and tracking of manual changes to determine information reliability.
- Provenance Chain Analysis - Evaluates data reliability by tracing the chain of custody and manual modifications from source to final set.
- Structural Dataset Error Detection - Identifies technical issues such as truncated spreadsheets and garbled text encoding caused by software limits.
- Dataset Validation Manuals - Provides a technical reference for spotting glitches like encoding errors and truncated files in raw data.
- Benford's Law Analysis - Identifies deceptive data patterns using red flags like p-hacking and Benford's Law failures.
- Population Bias Evaluations - Evaluates whether a dataset is representative by checking for non-random samples and biased population collection.
- Sampling Representativeness Assessments - Checks for non-random samples and biased populations to determine if a dataset is representative.
- Machine Learning - Reference for identifying and handling problematic datasets.
