Awesome GitHub RepositoriesData Quality

This project is an exploratory data analysis framework and profiling tool designed to generate comprehensive statistical reports from Pandas and Spark DataFrames. It functions as a data quality profiler that identifies missing values, duplicates, and high correlations within tabular datasets. The tool distinguishes itself through specialized capabilities for time-series analysis, extracting temporal statistics, seasonality, and auto-correlation plots. It also includes a dataset comparison utility to identify structural or content changes between different versions of a dataset. The analysis

Acest proiect este o colecție de framework-uri și pipeline-uri de big data, incluzând un framework de analiză Apache Hive, o platformă de analiză a datelor comportamentale, un motor de analiză predictivă și pipeline-uri de date în timp real. Oferă infrastructura necesară pentru construirea fluxurilor de lucru ETL (Extract, Transform, Load) pentru procesarea seturilor mari de date în vederea stocării distribuite și a analizei bazate pe SQL. Sistemul suportă implementări analitice diverse, cum ar fi un motor predictiv care utilizează regresia liniară pentru prognoza valorilor și o arhitectură în timp real care transmite datele prin message broker-e pentru raportare imediată. Include capabilități specializate pentru analiza comportamentului utilizatorilor, măsurarea performanței în e-commerce și analiza datelor de tranzit urban. Codul sursă acoperă o arie largă de inginerie și analiză a datelor, inclusiv curățarea și transformarea datelor, ingestia distribuită, procesarea fluxurilor bazată pe ferestre (window-based) și vizualizarea rezultatelor prin instrumente de business intelligence. De asemenea, permite calcularea unor metrici de business specifice, cum ar fi ratele de conversie, performanța monetizării și nivelurile de implicare a utilizatorilor.

Pandera is a data pipeline validation framework and statistical type validation tool. It functions as a library for defining and enforcing schemas on datasets to ensure data quality and consistency, specifically providing validation capabilities for Pandas dataframes. The project includes a schema inference tool that automates setup by analyzing existing dataset samples to generate validation schemas. It also serves as a synthetic data generator, creating artificial datasets based on predefined schemas to verify data-producing functions. The framework covers data engineering quality assuranc

Pimcore is an open-source data experience platform that serves as a unified framework for managing product information, digital assets, and customer data. It functions as an enterprise content management system and a master data management platform, providing a centralized source of truth for complex business information. The system is designed to support omnichannel delivery, enabling organizations to publish content and manage digital experiences across diverse platforms through both traditional and headless architectures. The platform distinguishes itself through a metadata-driven object m

Elementary OSS: dbt-native data observability

dbt-expectations is an extension package for dbt, inspired by the Great Expectations package for Python. The intent is to allow dbt users to deploy GE-like tests in their data warehouse directly from dbt, vs having to add another integration with their data warehouse.

Data breaks. Servers break. Your toolchain breaks. Ensure your data team is the first to know and the first to solve with visibility across and down your data estate. Save time with simple, fast data quality test generation and execution. Trust your data, tools, and systems from end to end.

The purpose of the dq tool is to make simple storing test results and visualisation of these in a BI dashboard.

A detective for your data. Zero-config data quality monitoring — works with dbt, Postgres, BigQuery, Snowflake. No YAML.