Awesome GitHub RepositoriesData Analysis

Pathway is a high-performance data processing framework designed for building unified batch and streaming pipelines. It functions as an orchestrator for complex data transformations, utilizing a differential dataflow engine to process updates incrementally. By treating static datasets and continuous event streams with identical logic, the platform ensures exactly-once processing semantics and consistent results across diverse data sources. The framework distinguishes itself through its specialized support for real-time artificial intelligence and retrieval-augmented generation. It features in

Pandas is a high-performance data analysis library that provides a comprehensive framework for manipulating, cleaning, and transforming structured datasets. It centers on labeled one-dimensional and two-dimensional data structures, allowing users to construct, filter, and reshape tabular information while performing complex arithmetic and logical operations. The library distinguishes itself through a sophisticated indexing engine that enables automatic data alignment during calculations and relational merges. By utilizing a block-based memory layout, it optimizes cache locality for vectorized

Polars is a high-performance columnar data processing library designed for efficient analytical workflows. It functions as a structured data library that organizes information into typed columns, utilizing the Apache Arrow memory format to enable zero-copy data sharing and cache-friendly, vectorized operations. The engine is built to handle large-scale tabular datasets, providing both local and distributed analytical runtimes that scale from single-machine environments to multi-node clusters. The project distinguishes itself through a sophisticated lazy query engine that constructs abstract e

This project is a data profiling and exploratory data analysis tool designed to generate automated quality reports for Pandas and Spark dataframes. It serves as a system for computing descriptive statistics, identifying correlations, and analyzing univariate and multivariate data patterns. The tool provides specialized capabilities for comparing different versions of datasets to identify changes in data quality and distributions. It includes a dedicated profiler for time-dependent data to extract statistical information such as seasonality and auto-correlation. The software covers a broad an

Ydata-profiling is an automated exploratory data analysis framework designed to generate comprehensive statistical reports and visual summaries from dataframes. It functions as a diagnostic tool for assessing data quality, identifying missing values, duplicates, and outliers, while providing a scalable engine for profiling massive datasets across distributed enterprise environments. The project distinguishes itself through its ability to handle large-scale data through distributed task orchestration and lazy stream processing, which minimizes memory overhead during complex computations. It in

Datasette is a tool for publishing and sharing SQLite databases as public websites. It functions as a data publishing system that provides searchable interfaces and JSON APIs to expose the contents of SQLite files. The project enables both server-side and client-side execution. It can operate as an API server or as a database browser that runs entirely within a web browser using WebAssembly, allowing for serverless database access. The system supports a variety of deployment strategies, including containerized images for cloud hosting and a local development server for testing. It includes c

Modin is a distributed dataframe library and parallel data processing engine designed to handle large datasets that exceed system memory. It functions as a distributed computing framework that parallelizes data manipulation tasks across multiple CPU cores or clusters to increase throughput and avoid memory errors. The project mirrors the Pandas API, allowing for the distribution of data workflows without changing core code logic. It utilizes a pluggable backend interface, which enables users to switch between different distributed execution engines to optimize performance based on available h

Gonum is a numerical computing library for the Go programming language, providing a collection of packages for scientific computing, linear algebra, statistics, and optimization. It functions as a framework for performing complex numerical computations and solving systems of linear equations. The project includes a dedicated graph analysis framework for modeling network graphs and solving connectivity and pathfinding problems. It also provides a statistical analysis toolkit for computing descriptive and inferential statistics and estimating mixture entropy. The library's capability surface c

Fathom is a privacy-focused website analytics server written in Go. It monitors website traffic and page views without collecting personal data or using intrusive cookies, providing a self-hosted alternative for traffic monitoring. The system utilizes a Preact-based dashboard interface for visualizing traffic patterns and reports. Data is persisted in a SQL database analytics store, with support for MySQL, PostgreSQL, and SQLite. The project covers the collection of visitor data via lightweight tracking snippets and the management of that data through a pluggable storage layer. It includes m

Ibis is a portable Python dataframe library and multi-backend query engine that provides a unified interface for executing data transformations across diverse compute engines. It functions as a Python SQL expression compiler and dialect transpiler, allowing users to define data logic once and execute it across cloud warehouses, embedded databases, and distributed clusters without rewriting code. The project distinguishes itself through a database backend abstraction that decouples transformation logic from the underlying execution engine. It enables polyglot data workflows by mixing raw SQL s

aws-sdk-pandas 是一个 Python 库，将 pandas 数据帧与 AWS 服务集成，充当云数据 ETL 工具和数据湖连接器。它提供了一个统一界面，用于在内存中数据帧与云存储、数据库和数据仓库之间移动和转换数据。 该项目作为分布式计算编排器脱颖而出，能够将基于 pandas 的工作负载提交到 EMR 集群和无服务器处理环境。它进一步专门通过 Ray 集群初始化来协调分布式数据处理，以处理超出单机内存的数据集。 该库涵盖了广泛的功能，包括 S3 的对象存储管理、Athena 和 Redshift 的 SQL 查询执行，以及与 NoSQL、图和时间序列数据库的集成。它还包括通过 Glue 目录进行元数据管理、OpenSearch 数据索引以及在 QuickSight 中管理商业智能资产的实用程序。 其他功能包括检索密钥、分析 CloudWatch 日志以及管理数据质量规则集。

Gota: DataFrames and data wrangling in Go (Golang)

DATAGEN is a powerful brand name that represents our vision of leveraging artificial intelligence technology for data generation and analysis. The name combines "DATA" and "GEN"(generation), perfectly embodying the core functionality of this project - automated data analysis and research through…

A distributed, fault-tolerant pipeline for observability data

DataFrames for Go: For statistics, machine-learning, and data manipulation/exploration

A Clojure high performance data processing system

Map-Reduce for Clojure

Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.

AI Powered Data Agent framework, a comprehensive solution for data analysis, engineering, and visualization.

A library of statistical distribution sampling and transducing functions