Awesome GitHub RepositoriesData Quality and Validation

Pydantic is a data validation and serialization library that enforces schema constraints and performs type conversion on complex data structures. It utilizes standard Python type annotations to define data models, allowing developers to establish structured schemas that automatically enforce business rules and constraints without the need for custom domain-specific languages. The library distinguishes itself by transforming high-level model definitions into optimized code during initialization to minimize runtime overhead. It supports recursive validation for nested data structures and employ

Ludwig is a multimodal machine learning platform and low-code framework designed for building, training, and deploying neural networks. It enables the construction of models that process text, images, audio, and tabular data through a unified interface using declarative configuration files rather than custom code. The system features a specialized low-code framework for large language models, supporting supervised fine-tuning, preference alignment, and a constrained decoding tool to force structured data output via logit extraction. It also includes an automated model architecture search to i

PyOD is a Python anomaly detection library used to identify outliers in tabular, time series, graph, text, and image data. It provides a collection of algorithms for detecting anomalous data points and includes a unified detector interface that standardizes input and output signatures across its available detection algorithms. The project features a multi-modal outlier detector for identifying anomalies across diverse formats including unstructured text and images, as well as a specialized toolkit for graph-based and time-series anomaly detection. It includes an ensemble framework for combini

SeaTunnel is a distributed data integration engine designed to synchronize structured and unstructured data across diverse sources and sinks. It functions as a multi-engine execution framework that can run data integration tasks across different distributed computing backends to optimize workload performance. The project is distinguished by a visual data pipeline designer for configuring workflows without manual code and a specialized change data capture tool for streaming incremental database updates. It also includes an enrichment pipeline that integrates large language models and embedding

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

Luban is a game configuration toolchain designed to convert spreadsheet-based data into optimized binary formats and type-safe source code for multiple languages. It functions as a comprehensive suite for configuration validation, data serialization pipelines, and code generation to ensure data consistency across different platforms. The system features a multi-language code generator that produces strongly-typed data classes from schemas, eliminating the need for reflection. It includes a localization manager for exporting translated text and assets with locale-specific patching, and a seria

Pandera is a data pipeline validation framework and statistical type validation tool. It functions as a library for defining and enforcing schemas on datasets to ensure data quality and consistency, specifically providing validation capabilities for Pandas dataframes. The project includes a schema inference tool that automates setup by analyzing existing dataset samples to generate validation schemas. It also serves as a synthetic data generator, creating artificial datasets based on predefined schemas to verify data-producing functions. The framework covers data engineering quality assuranc

Lightweight, extensible data validation library for Python

Data engineering practice repository providing tutorials, distributed processing engines, and Python data pipeline automation scripts. The system encompasses automated data validation, distributed compute aggregation, embedded columnar querying, lazy evaluation planning, partitioned storage export, and cloud storage retrieval. The capability surface covers cloud integration and storage, data engineering and pipelines, data processing and analytics, data quality and testing, database and storage, file management, and monitoring and observability.

Algorithms for outlier, adversarial and drift detection

هذا المشروع عبارة عن منصة مفتوحة المصدر لإدارة علاقات العملاء (CRM) تعمل كإطار عمل لتطوير التطبيقات منخفضة الكود (low-code). توفر واجهة موحدة لتتبع خطوط أنابيب المبيعات، وإدارة تفاعلات العملاء، وأتمتة توجيه العملاء المحتملين. تم بناء المنصة لتعمل كأداة لأتمتة عمليات الأعمال، مما يسمح للمستخدمين بتحديد هياكل بيانات وسير عمل مخصصة لتبسيط المهام التشغيلية. يتميز النظام بمعماريته المدفوعة بالبيانات الوصفية، والتي تتيح توليد النماذج الديناميكي ونمذجة المستندات العلائقية. ومن خلال استخدام حقن نصوص الخادم (server-side script injection) وكتابة نصوص النماذج المخصصة، يمكن للمستخدمين فرض منطق عمل معقد وقواعد التحقق من البيانات مباشرة داخل التطبيق. كما يندمج النظام مع تخطيط موارد المؤسسات وأنظمة الاتصال الخارجية، مما يركز السجلات المالية وقنوات المراسلة في لوحة تحكم إدارة واحدة. تدعم المنصة نطاقاً واسعاً من القدرات التشغيلية، بما في ذلك تعيين العملاء المحتملين آلياً، وفرض اتفاقيات مستوى الخدمة، والاتصال متعدد القنوات. يمكن للمستخدمين تنظيم البيانات من خلال طرق عرض وتصورات مخصصة، بينما يسهل إطار العمل الأساسي تعاون الفريق من خلال الملاحظات المشتركة وسجلات المهام. تم تصميم النظام للنشر في بيئات الحاويات، مما يضمن أداء متسقاً عبر البنية التحتية الخاصة أو السحابية.

Tools for exploratory data analysis in Python