Awesome GitHub RepositoriesLarge-Scale Data Computation

Apache Spark is a unified distributed data processing engine designed for large-scale data analysis and computation graphs. It functions as a distributed machine learning framework, a graph processing system, a real-time stream processor, and a SQL analytics engine. The system enables the execution of distributed SQL querying, large-scale graph analysis, and real-time stream analytics across clusters of machines. It also provides a scalable environment for implementing machine learning algorithms and predictive model development on massive datasets. The engine incorporates relational query e

Gensim is an unsupervised natural language processing toolkit designed for topic modeling, word embedding training, and the processing of large-scale text corpora. It provides a framework for discovering latent themes and semantic structures in text without the need for labeled data. The toolkit is distinguished by its ability to handle datasets that exceed system memory through iterator-based data streaming from disk. It also supports distributed model training, allowing complex modeling tasks to be executed across computer clusters. The library covers a broad range of analysis capabilities

Hadoop is a big data infrastructure suite and distributed data processing framework designed to store and process massive datasets across clusters of computers. It consists of a distributed storage system for managing large files across multiple nodes and a parallel computing engine for processing data across a distributed cluster. The framework implements a distributed file system to ensure fault tolerance and high throughput, paired with a programming model that processes large datasets in parallel. It manages the underlying hardware and software environment required for distributed big dat

Apache Druid is a real-time analytics database and distributed columnar time-series store designed for sub-second analytical queries. It functions as a data platform featuring a distributed SQL query engine and a real-time data ingestion system for moving historical and streaming data from external sources. The system is distinguished by its ability to provide low-latency analytics under high concurrency to power operational dashboards. It implements a Kerberos-secured environment for user authentication and employs a shared-nothing cluster architecture to enable horizontal scaling. The plat

Azure Docs is the official technical documentation repository for Microsoft Azure, the cloud computing platform. It provides comprehensive guidance on the full spectrum of Azure services, covering everything from core infrastructure components like virtual machines, Kubernetes clusters, and serverless computing to platform services for AI, machine learning, data analytics, and storage. The documentation details how to provision, manage, and govern cloud resources at scale, including policy enforcement, identity management, and cost optimization. The documentation distinguishes Azure through i

Modin is a distributed dataframe library and parallel data processing engine designed to handle large datasets that exceed system memory. It functions as a distributed computing framework that parallelizes data manipulation tasks across multiple CPU cores or clusters to increase throughput and avoid memory errors. The project mirrors the Pandas API, allowing for the distribution of data workflows without changing core code logic. It utilizes a pluggable backend interface, which enables users to switch between different distributed execution engines to optimize performance based on available h

Boto3 is the AWS SDK for Python, providing a programmatic interface for managing and automating AWS cloud infrastructure and services. It serves as a cloud management API client and resource manager for provisioning, configuring, and scaling virtual servers, databases, and storage. The library enables the implementation of infrastructure-as-code through declarative templates and scripts, allowing for the deployment of identical resource stacks across multiple accounts and geographic regions. It also provides a framework for coordinating distributed workflows, serverless functions, and contain

tsfresh is an automated feature engineering tool and library designed to extract statistical characteristics from raw time series data. It transforms sequential data into tabular datasets, converting time series into a flat format where each row represents a unique entity and columns represent extracted features. The project distinguishes itself through a parallel data processing framework that distributes heavy computational workloads across multiple CPU cores. It also implements hypothesis-based feature selection to identify the most predictive characteristics and filter out irrelevant ones

Spring AI is an application framework for Java that provides a portable, fluent API for integrating AI models, tools, and vector stores into applications. It wraps multiple AI providers behind a common interface, allowing developers to switch between chat, embedding, image, and speech models without changing application code. The framework includes a chainable chat client API similar to WebClient or RestClient, supports both synchronous and streaming interactions, and offers structured output conversion that transforms unstructured AI responses into strongly-typed Java objects. The framework

LanceDB is a vector database and columnar data store designed to function as a versioned dataset manager and vector search engine. It serves as a high-performance backend for indexing and retrieving high-dimensional embeddings, providing the foundation for machine learning data pipelines. The system distinguishes itself through a combination of cloud-native object storage and immutable version tracking, allowing for data time-travel and reproducible AI experiments. It integrates hybrid search capabilities, merging dense vector similarity with BM25 full-text search and SQL-like scalar filters

This project is an educational resource and a collection of instructional materials for performing data manipulation and statistical analysis using Python. It provides a comprehensive set of guides and code examples for using the Pandas, NumPy, and Matplotlib libraries to analyze structured data. The resource includes a dedicated guide for reshaping, cleaning, and aggregating tabular data and time series via Pandas, alongside a reference for high-performance vectorized operations and linear algebra using NumPy. It also features tutorials for creating publication-quality charts, distribution p

Vaex is a high-performance Apache Arrow DataFrame library and out-of-core data processing engine designed to handle billion-row tabular datasets in Python. It functions as a lazy evaluation framework that defers computations and transformations until results are required, enabling the processing of datasets that exceed available system RAM by mapping files directly from disk. The project distinguishes itself as a tool for big data visualization and exploration, specifically integrated for use within interactive notebooks. It provides specialized capabilities for machine learning feature engin

The 1BRC (One Billion Row Challenge) is a Java performance benchmarking exercise that processes one billion temperature records from a text file to compute the minimum, mean, and maximum temperature per weather station. At its core, it is a large-scale data aggregation challenge designed to test how efficiently a Java program can parse and aggregate structured data from a plain text file, serving as both a programming exercise and a benchmark for Java performance optimization. The project distinguishes itself through a collection of performance-oriented architectural patterns for high-through

Featuretools is an automated feature engineering library and data transformation framework written in Python. It automatically generates machine learning feature vectors from multi-table datasets by applying synthesis patterns to relational and timestamped data. The system functions as a distributed feature synthesis engine, allowing the process of creating feature vectors to scale across multiple cores or clusters to handle large-scale datasets. The library supports the synthesis of multi-table datasets, time series feature generation, and the creation of custom machine learning primitives

Featuretools is a Python data science library and automated feature engineering framework designed to create predictive features from multiple related datasets. It automates the data preparation and transformation steps required for machine learning models through deep feature synthesis. The library enables the automatic generation of comprehensive feature tables by applying recursive transformations to relational data. It supports the transformation of unstructured text into structured numeric features and allows users to define custom primitives to extend the synthesis process with specific

h2o-3 is a distributed machine learning platform and automated machine learning framework designed for training and deploying predictive models using distributed in-memory computing. It functions as a deep learning framework and a distributed model scoring engine, capable of operating as a Kubernetes ML cluster to process large datasets in parallel. The platform distinguishes itself through automated machine learning capabilities that automatically select the best algorithms and hyperparameters to optimize model performance. It provides specialized deep learning toolkits for tasks including i

oneTBB एक C++ पैरेललिज़्म लाइब्रेरी और फ्रेमवर्क है जिसे एप्लिकेशन में मल्टी-कोर पैरेललिज़्म जोड़ने के लिए डिज़ाइन किया गया है। यह एक टास्क-आधारित पैरेललिज़्म मॉडल प्रदान करता है जो मैनुअल थ्रेड मैनेजमेंट की आवश्यकता को समाप्त करने के लिए उपलब्ध हार्डवेयर कोर पर तार्किक कंप्यूटेशनल कार्यों को मैप करता है। यह लाइब्रेरी एक मल्टी-कोर स्केलिंग टूल के रूप में कार्य करती है, जो पोर्टेबल प्रदर्शन के लिए प्रोसेसर पर डेटा-पैरेलल ऑपरेशन्स को स्केल करने के लिए जेनेरिक टेम्प्लेट का उपयोग करती है। यह सुनिश्चित करने के लिए कि कंप्यूटेशनल वर्कलोड हार्डवेयर संसाधनों पर वितरित किए जाते हैं, एक टास्क-आधारित फ्रेमवर्क का उपयोग करती है। यह प्रोजेक्ट शेयर्ड मेमोरी पैरेललिज़्म, मल्टी-कोर टास्क शेड्यूलिंग और डेटा पैरेललिज़्म स्केलिंग को कवर करता है। यह रनटाइम पर कोर के बीच काम के वितरण को प्रबंधित करने के लिए वर्क-स्टीलिंग टास्क शेड्यूलर, रिकर्सिव रेंज स्प्लिटिंग और डायनामिक लोड बैलेंसिंग का उपयोग करता है।

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

Smile is a comprehensive JVM machine learning library and statistical computing toolkit. It provides a suite of algorithms for classification, regression, and clustering, implemented natively for Java, Scala, and Kotlin. The project also functions as a deep learning framework, a natural language processing library, and an inference engine for large language models. The library distinguishes itself through GPU acceleration via LibTorch bindings and support for the ONNX model interchange format. It includes specialized capabilities for large language model inference, featuring Byte-Pair Encodin

River ऑनलाइन मशीन लर्निंग के लिए एक Python फ़्रेमवर्क है, जिसे स्ट्रीमिंग डेटा पर मॉडल को ट्रेन और इवैल्यूएट करने के लिए डिज़ाइन किया गया है। यह एक बार में एक ऑब्ज़र्वेशन के साथ मॉडल पैरामीटर्स को अपडेट करके इंक्रीमेंटल लर्निंग को सक्षम बनाता है, जिससे मेमोरी में पूर्ण ट्रेनिंग डेटासेट को स्टोर करने की आवश्यकता समाप्त हो जाती है। यह लाइब्रेरी एक डेडिकेटेड कॉन्सेप्ट ड्रिफ्ट डिटेक्शन सिस्टम के माध्यम से खुद को अलग करती है जो मॉडल एडॉप्टेशन को ट्रिगर करने के लिए डेटा वितरण में परिवर्तनों की निगरानी करता है। यह एक प्रोग्रेसिव वैलिडेशन फ़्रेमवर्क भी प्रदान करती है जो ट्रेनिंग के लिए उपयोग करने से पहले सैंपल्स पर मॉडल का परीक्षण करके रीयल-टाइम डिप्लॉयमेंट को सिम्युलेट करता है। यह सिस्टम रीयल-टाइम फ़ीचर इंजीनियरिंग, टाइम सीरीज़ फ़ोरकास्टिंग और ऑनलाइन एनोमली डिटेक्शन सहित स्ट्रीमिंग क्षमताओं की एक विस्तृत श्रृंखला को कवर करता है। यह इंक्रीमेंटल क्लस्टरिंग और डिसीजन ट्री के माध्यम से अनसुपरवाइज़्ड लर्निंग, साथ ही मॉडल चयन के लिए एन्सेम्बल एग्रीगेशन और बैंडिट नीतियों का समर्थन करता है। इस प्रोजेक्ट में CSV फ़ाइलों और API जैसे स्रोतों से स्ट्रीमिंग डेटा इनजेशन के लिए यूटिलिटीज़, साथ ही रनिंग आँकड़ों और मेमोरी-कुशल डेटा स्केच की गणना करने के लिए टूल शामिल हैं।