Awesome GitHub RepositoriesData Analysis

Pathway is a high-performance data processing framework designed for building unified batch and streaming pipelines. It functions as an orchestrator for complex data transformations, utilizing a differential dataflow engine to process updates incrementally. By treating static datasets and continuous event streams with identical logic, the platform ensures exactly-once processing semantics and consistent results across diverse data sources. The framework distinguishes itself through its specialized support for real-time artificial intelligence and retrieval-augmented generation. It features in

Pandas is a high-performance data analysis library that provides a comprehensive framework for manipulating, cleaning, and transforming structured datasets. It centers on labeled one-dimensional and two-dimensional data structures, allowing users to construct, filter, and reshape tabular information while performing complex arithmetic and logical operations. The library distinguishes itself through a sophisticated indexing engine that enables automatic data alignment during calculations and relational merges. By utilizing a block-based memory layout, it optimizes cache locality for vectorized

Polars is a high-performance columnar data processing library designed for efficient analytical workflows. It functions as a structured data library that organizes information into typed columns, utilizing the Apache Arrow memory format to enable zero-copy data sharing and cache-friendly, vectorized operations. The engine is built to handle large-scale tabular datasets, providing both local and distributed analytical runtimes that scale from single-machine environments to multi-node clusters. The project distinguishes itself through a sophisticated lazy query engine that constructs abstract e

This project is a data profiling and exploratory data analysis tool designed to generate automated quality reports for Pandas and Spark dataframes. It serves as a system for computing descriptive statistics, identifying correlations, and analyzing univariate and multivariate data patterns. The tool provides specialized capabilities for comparing different versions of datasets to identify changes in data quality and distributions. It includes a dedicated profiler for time-dependent data to extract statistical information such as seasonality and auto-correlation. The software covers a broad an

Ydata-profiling is an automated exploratory data analysis framework designed to generate comprehensive statistical reports and visual summaries from dataframes. It functions as a diagnostic tool for assessing data quality, identifying missing values, duplicates, and outliers, while providing a scalable engine for profiling massive datasets across distributed enterprise environments. The project distinguishes itself through its ability to handle large-scale data through distributed task orchestration and lazy stream processing, which minimizes memory overhead during complex computations. It in

Datasette is a tool for publishing and sharing SQLite databases as public websites. It functions as a data publishing system that provides searchable interfaces and JSON APIs to expose the contents of SQLite files. The project enables both server-side and client-side execution. It can operate as an API server or as a database browser that runs entirely within a web browser using WebAssembly, allowing for serverless database access. The system supports a variety of deployment strategies, including containerized images for cloud hosting and a local development server for testing. It includes c

Modin is a distributed dataframe library and parallel data processing engine designed to handle large datasets that exceed system memory. It functions as a distributed computing framework that parallelizes data manipulation tasks across multiple CPU cores or clusters to increase throughput and avoid memory errors. The project mirrors the Pandas API, allowing for the distribution of data workflows without changing core code logic. It utilizes a pluggable backend interface, which enables users to switch between different distributed execution engines to optimize performance based on available h

Gonum is a numerical computing library for the Go programming language, providing a collection of packages for scientific computing, linear algebra, statistics, and optimization. It functions as a framework for performing complex numerical computations and solving systems of linear equations. The project includes a dedicated graph analysis framework for modeling network graphs and solving connectivity and pathfinding problems. It also provides a statistical analysis toolkit for computing descriptive and inferential statistics and estimating mixture entropy. The library's capability surface c

Fathom is a privacy-focused website analytics server written in Go. It monitors website traffic and page views without collecting personal data or using intrusive cookies, providing a self-hosted alternative for traffic monitoring. The system utilizes a Preact-based dashboard interface for visualizing traffic patterns and reports. Data is persisted in a SQL database analytics store, with support for MySQL, PostgreSQL, and SQLite. The project covers the collection of visitor data via lightweight tracking snippets and the management of that data through a pluggable storage layer. It includes m

Ibis is a portable Python dataframe library and multi-backend query engine that provides a unified interface for executing data transformations across diverse compute engines. It functions as a Python SQL expression compiler and dialect transpiler, allowing users to define data logic once and execute it across cloud warehouses, embedded databases, and distributed clusters without rewriting code. The project distinguishes itself through a database backend abstraction that decouples transformation logic from the underlying execution engine. It enables polyglot data workflows by mixing raw SQL s

aws-sdk-pandas est une bibliothèque Python qui intègre les dataframes pandas avec les services AWS, agissant comme un outil ETL de données cloud et un connecteur de lac de données. Elle fournit une interface unifiée pour déplacer et transformer les données entre des dataframes en mémoire et le stockage cloud, les bases de données et les entrepôts de données. Le projet se distingue comme un orchestrateur de calcul distribué capable de soumettre des charges de travail basées sur pandas à des clusters EMR et des environnements de traitement sans serveur. Il se spécialise davantage dans la coordination du traitement de données distribué via l'initialisation de clusters Ray pour gérer des jeux de données qui dépassent la mémoire d'une seule machine. La bibliothèque couvre un large éventail de capacités, incluant la gestion du stockage d'objets pour S3, l'exécution de requêtes SQL pour Athena et Redshift, et l'intégration avec des bases de données NoSQL, graphes et séries temporelles. Elle inclut également des utilitaires pour la gestion des métadonnées via le catalogue Glue, l'indexation de données OpenSearch et la gestion des actifs de business intelligence dans QuickSight. La fonctionnalité supplémentaire inclut la récupération de secrets, l'analyse des journaux CloudWatch et la gestion des ensembles de règles de qualité des données.

Gota: DataFrames and data wrangling in Go (Golang)

DATAGEN is a powerful brand name that represents our vision of leveraging artificial intelligence technology for data generation and analysis. The name combines "DATA" and "GEN"(generation), perfectly embodying the core functionality of this project - automated data analysis and research through…

A distributed, fault-tolerant pipeline for observability data

DataFrames for Go: For statistics, machine-learning, and data manipulation/exploration

A Clojure high performance data processing system

Map-Reduce for Clojure

Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.

AI Powered Data Agent framework, a comprehensive solution for data analysis, engineering, and visualization.

A library of statistical distribution sampling and transducing functions