Awesome GitHub RepositoriesData Analysis

Pathway is a high-performance data processing framework designed for building unified batch and streaming pipelines. It functions as an orchestrator for complex data transformations, utilizing a differential dataflow engine to process updates incrementally. By treating static datasets and continuous event streams with identical logic, the platform ensures exactly-once processing semantics and consistent results across diverse data sources. The framework distinguishes itself through its specialized support for real-time artificial intelligence and retrieval-augmented generation. It features in

Pandas is a high-performance data analysis library that provides a comprehensive framework for manipulating, cleaning, and transforming structured datasets. It centers on labeled one-dimensional and two-dimensional data structures, allowing users to construct, filter, and reshape tabular information while performing complex arithmetic and logical operations. The library distinguishes itself through a sophisticated indexing engine that enables automatic data alignment during calculations and relational merges. By utilizing a block-based memory layout, it optimizes cache locality for vectorized

Polars is a high-performance columnar data processing library designed for efficient analytical workflows. It functions as a structured data library that organizes information into typed columns, utilizing the Apache Arrow memory format to enable zero-copy data sharing and cache-friendly, vectorized operations. The engine is built to handle large-scale tabular datasets, providing both local and distributed analytical runtimes that scale from single-machine environments to multi-node clusters. The project distinguishes itself through a sophisticated lazy query engine that constructs abstract e

This project is a data profiling and exploratory data analysis tool designed to generate automated quality reports for Pandas and Spark dataframes. It serves as a system for computing descriptive statistics, identifying correlations, and analyzing univariate and multivariate data patterns. The tool provides specialized capabilities for comparing different versions of datasets to identify changes in data quality and distributions. It includes a dedicated profiler for time-dependent data to extract statistical information such as seasonality and auto-correlation. The software covers a broad an

Ydata-profiling is an automated exploratory data analysis framework designed to generate comprehensive statistical reports and visual summaries from dataframes. It functions as a diagnostic tool for assessing data quality, identifying missing values, duplicates, and outliers, while providing a scalable engine for profiling massive datasets across distributed enterprise environments. The project distinguishes itself through its ability to handle large-scale data through distributed task orchestration and lazy stream processing, which minimizes memory overhead during complex computations. It in

Datasette is a tool for publishing and sharing SQLite databases as public websites. It functions as a data publishing system that provides searchable interfaces and JSON APIs to expose the contents of SQLite files. The project enables both server-side and client-side execution. It can operate as an API server or as a database browser that runs entirely within a web browser using WebAssembly, allowing for serverless database access. The system supports a variety of deployment strategies, including containerized images for cloud hosting and a local development server for testing. It includes c

Modin is a distributed dataframe library and parallel data processing engine designed to handle large datasets that exceed system memory. It functions as a distributed computing framework that parallelizes data manipulation tasks across multiple CPU cores or clusters to increase throughput and avoid memory errors. The project mirrors the Pandas API, allowing for the distribution of data workflows without changing core code logic. It utilizes a pluggable backend interface, which enables users to switch between different distributed execution engines to optimize performance based on available h

Gonum is a numerical computing library for the Go programming language, providing a collection of packages for scientific computing, linear algebra, statistics, and optimization. It functions as a framework for performing complex numerical computations and solving systems of linear equations. The project includes a dedicated graph analysis framework for modeling network graphs and solving connectivity and pathfinding problems. It also provides a statistical analysis toolkit for computing descriptive and inferential statistics and estimating mixture entropy. The library's capability surface c

Fathom is a privacy-focused website analytics server written in Go. It monitors website traffic and page views without collecting personal data or using intrusive cookies, providing a self-hosted alternative for traffic monitoring. The system utilizes a Preact-based dashboard interface for visualizing traffic patterns and reports. Data is persisted in a SQL database analytics store, with support for MySQL, PostgreSQL, and SQLite. The project covers the collection of visitor data via lightweight tracking snippets and the management of that data through a pluggable storage layer. It includes m

Ibis is a portable Python dataframe library and multi-backend query engine that provides a unified interface for executing data transformations across diverse compute engines. It functions as a Python SQL expression compiler and dialect transpiler, allowing users to define data logic once and execute it across cloud warehouses, embedded databases, and distributed clusters without rewriting code. The project distinguishes itself through a database backend abstraction that decouples transformation logic from the underlying execution engine. It enables polyglot data workflows by mixing raw SQL s

aws-sdk-pandas एक Python लाइब्रेरी है जो pandas डेटाफ्रेम्स को AWS सेवाओं के साथ एकीकृत करती है, जो एक क्लाउड डेटा ETL टूल और डेटा लेक कनेक्टर के रूप में कार्य करती है। यह इन-मेमोरी डेटाफ्रेम्स और क्लाउड स्टोरेज, डेटाबेस और डेटा वेयरहाउस के बीच डेटा को स्थानांतरित और बदलने के लिए एक एकीकृत इंटरफेस प्रदान करती है। प्रोजेक्ट एक डिस्ट्रीब्यूटेड कंप्यूट ऑर्केस्ट्रेटर के रूप में खुद को अलग करता है जो एक मशीन की मेमोरी से अधिक डेटासेट्स को संभालने के लिए EMR क्लस्टर्स और सर्वरलेस प्रोसेसिंग वातावरण में pandas-आधारित वर्कफ़्लो सबमिट करने में सक्षम है। यह एक मशीन की मेमोरी से अधिक डेटासेट्स को संभालने के लिए Ray क्लस्टर इनिशियलाइज़ेशन के माध्यम से डिस्ट्रीब्यूटेड डेटा प्रोसेसिंग के समन्वय में और विशेषज्ञता रखता है। लाइब्रेरी क्षमताओं की एक विस्तृत श्रृंखला को कवर करती है, जिसमें S3 के लिए ऑब्जेक्ट स्टोरेज प्रबंधन, Athena और Redshift के लिए SQL क्वेरी निष्पादन, और NoSQL, ग्राफ और टाइम-सीरीज डेटाबेस के साथ एकीकरण शामिल है। इसमें Glue कैटलॉग के माध्यम से मेटाडेटा प्रबंधन, OpenSearch डेटा इंडेक्सिंग, और QuickSight में बिजनेस इंटेलिजेंस एसेट्स का प्रबंधन करने के लिए यूटिलिटीज भी शामिल हैं। अतिरिक्त कार्यक्षमता में सीक्रेट्स प्राप्त करना, CloudWatch लॉग्स का विश्लेषण करना और डेटा गुणवत्ता नियम सेट प्रबंधित करना शामिल है।

Gota: DataFrames and data wrangling in Go (Golang)

DATAGEN is a powerful brand name that represents our vision of leveraging artificial intelligence technology for data generation and analysis. The name combines "DATA" and "GEN"(generation), perfectly embodying the core functionality of this project - automated data analysis and research through…

A distributed, fault-tolerant pipeline for observability data

DataFrames for Go: For statistics, machine-learning, and data manipulation/exploration

A Clojure high performance data processing system

Map-Reduce for Clojure

Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.

AI Powered Data Agent framework, a comprehensive solution for data analysis, engineering, and visualization.

A library of statistical distribution sampling and transducing functions