Awesome GitHub RepositoriesData Quality and Validation

Pydantic is a data validation and serialization library that enforces schema constraints and performs type conversion on complex data structures. It utilizes standard Python type annotations to define data models, allowing developers to establish structured schemas that automatically enforce business rules and constraints without the need for custom domain-specific languages. The library distinguishes itself by transforming high-level model definitions into optimized code during initialization to minimize runtime overhead. It supports recursive validation for nested data structures and employ

Ludwig is a multimodal machine learning platform and low-code framework designed for building, training, and deploying neural networks. It enables the construction of models that process text, images, audio, and tabular data through a unified interface using declarative configuration files rather than custom code. The system features a specialized low-code framework for large language models, supporting supervised fine-tuning, preference alignment, and a constrained decoding tool to force structured data output via logit extraction. It also includes an automated model architecture search to i

PyOD is a Python anomaly detection library used to identify outliers in tabular, time series, graph, text, and image data. It provides a collection of algorithms for detecting anomalous data points and includes a unified detector interface that standardizes input and output signatures across its available detection algorithms. The project features a multi-modal outlier detector for identifying anomalies across diverse formats including unstructured text and images, as well as a specialized toolkit for graph-based and time-series anomaly detection. It includes an ensemble framework for combini

SeaTunnel is a distributed data integration engine designed to synchronize structured and unstructured data across diverse sources and sinks. It functions as a multi-engine execution framework that can run data integration tasks across different distributed computing backends to optimize workload performance. The project is distinguished by a visual data pipeline designer for configuring workflows without manual code and a specialized change data capture tool for streaming incremental database updates. It also includes an enrichment pipeline that integrates large language models and embedding

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

Luban est une chaîne d'outils de configuration de jeu conçue pour convertir des données basées sur des feuilles de calcul en formats binaires optimisés et en code source typé pour plusieurs langages. Il fonctionne comme une suite complète pour la validation de configuration, les pipelines de sérialisation de données et la génération de code afin d'assurer la cohérence des données à travers différentes plateformes. Le système dispose d'un générateur de code multi-langage qui produit des classes de données fortement typées à partir de schémas, éliminant le besoin de réflexion. Il inclut un gestionnaire de localisation pour exporter du texte traduit et des assets avec des correctifs spécifiques à la locale, ainsi qu'un pipeline de sérialisation qui transforme les fichiers sources structurés en binaires pour un transfert réseau efficace. La chaîne d'outils fournit un moteur de validation de configuration pour effectuer des vérifications d'intégrité référentielle et de vérification des chemins de ressources. Elle supporte en outre la modélisation de données complexe via un système de types orienté objet qui permet l'héritage de données et des structures imbriquées au sein des fichiers de configuration.

Pandera is a data pipeline validation framework and statistical type validation tool. It functions as a library for defining and enforcing schemas on datasets to ensure data quality and consistency, specifically providing validation capabilities for Pandas dataframes. The project includes a schema inference tool that automates setup by analyzing existing dataset samples to generate validation schemas. It also serves as a synthetic data generator, creating artificial datasets based on predefined schemas to verify data-producing functions. The framework covers data engineering quality assuranc

Lightweight, extensible data validation library for Python

Data engineering practice repository providing tutorials, distributed processing engines, and Python data pipeline automation scripts. The system encompasses automated data validation, distributed compute aggregation, embedded columnar querying, lazy evaluation planning, partitioned storage export, and cloud storage retrieval. The capability surface covers cloud integration and storage, data engineering and pipelines, data processing and analytics, data quality and testing, database and storage, file management, and monitoring and observability.

Algorithms for outlier, adversarial and drift detection

This project is an open-source customer relationship management platform that functions as a low-code application development framework. It provides a unified interface for tracking sales pipelines, managing customer interactions, and automating lead routing. The platform is built to serve as a business process automation tool, allowing users to define custom data structures and workflows to streamline operational tasks. The system distinguishes itself through its metadata-driven architecture, which enables dynamic form generation and relational document modeling. By utilizing server-side scr

Tools for exploratory data analysis in Python