Quartz/bad-data-guide
Bad Data Guide
هذا المشروع عبارة عن مجموعة من المواد المرجعية والمبادئ التوجيهية لتنفيذ أطر تدقيق البيانات. يعمل كدليل مرجعي لجودة البيانات ودليل للتحقق من مجموعات البيانات لتحديد الأخطاء الهيكلية والإحصائية الشائعة في مجموعات البيانات.
يوفر المشروع قاعدة معرفية منظمة لتنظيف البيانات، تتميز بكتالوج لأخطاء البيانات الواقعية واستراتيجيات عملية لاكتشافها وحلها. يتضمن أطراً محددة لتقييم مصدر البيانات وموثوقية المعلومات المجمعة.
تغطي المادة مجموعة واسعة من قدرات تحليل البيانات، بما في ذلك التحقق من السلامة الإحصائية لاكتشاف التلاعب، وتقييمات صلاحية أخذ العينات لتحديد تحيز السكان، وطرق اكتشاف الأخطاء الهيكلية مثل مشاكل الترميز. كما يصف عمليات استعادة المعلومات الجدولية من المستندات المرئية من خلال التعرف الضوئي على الحروف (OCR).
Features
- Data Audit Frameworks - Implements a comprehensive framework for evaluating sampling validity, provenance, and aggregation reliability in datasets.
- Dataset Quality Auditing - Identifies common errors and unreliable patterns in real-world datasets to prevent mistakes during the analytical process.
- Data Auditing - Identifies patterns of unreliable data such as suspicious default values and inconsistent spelling.
- Data Cleaning Procedures - Provides workflows for finding and resolving structural errors and inconsistent values using expert-led mitigation strategies.
- Data Error Mitigation Strategies - Provides suggested solutions and mitigation strategies for data errors, categorized by the expert best equipped to fix them.
- Data Quality Profilers - References common real-world data errors and applies methods to resolve or mitigate those issues.
- Statistical Integrity Validators - Provides a framework for identifying analytical errors and deceptive patterns like p-hacking to ensure statistical integrity.
- Data Quality Guides - Provides a comprehensive catalog of real-world data errors with practical strategies for detection and resolution.
- Data Quality Taxonomies - Organizes data quality issues into a structured hierarchy based on their root cause and impact.
- Statistical Anomaly Heuristics - Implements rule-based heuristic checks to identify statistical anomalies and population sampling biases.
- Integrity Signature Matching - Identifies unreliable data by matching observed values against known signatures of software errors and human bias.
- Data Extraction and OCR - Extracts and recovers tabular information from PDFs and scanned images using OCR tools.
- OCR Document Parsers - Utilizes OCR-driven document parsing to extract structured tabular information from images and scanned PDFs.
- Document Data Extraction - Recovers structured tabular data from PDFs and scanned images using optical character recognition processes.
- Data Provenance Frameworks - Enables investigation of collection processes and tracking of manual changes to determine information reliability.
- Provenance Chain Analysis - Evaluates data reliability by tracing the chain of custody and manual modifications from source to final set.
- Structural Dataset Error Detection - Identifies technical issues such as truncated spreadsheets and garbled text encoding caused by software limits.
- Dataset Validation Manuals - Provides a technical reference for spotting glitches like encoding errors and truncated files in raw data.
- Benford's Law Analysis - Identifies deceptive data patterns using red flags like p-hacking and Benford's Law failures.
- Population Bias Evaluations - Evaluates whether a dataset is representative by checking for non-random samples and biased population collection.
- Sampling Representativeness Assessments - Checks for non-random samples and biased populations to determine if a dataset is representative.
- Machine Learning - Reference for identifying and handling problematic datasets.
