Awesome GitHub RepositoriesMapReduce Processing Engines

Redisson is a Java client library for Redis and Valkey that provides a distributed data structure library, a distributed lock manager, and a distributed MapReduce framework. It enables application instances in a cluster to share state through thread-safe collections and objects. The project implements a JCache compliant caching layer for standardized data storage and retrieval. It also functions as a probabilistic data store, providing memory-efficient structures such as Bloom filters and HyperLogLog for high-volume data membership testing. The library covers distributed state management usi

This project serves as a comprehensive technical reference for the architecture and design of data-intensive applications. It provides a structured analysis of the fundamental principles required to build reliable, scalable, and maintainable software systems, covering the core trade-offs inherent in modern data infrastructure. The repository explores the mechanics of distributed data management, including strategies for replication, partitioning, and achieving consensus across multiple nodes. It details the design of storage engines, indexing techniques, and transaction management models, whi

PouchDB is a JavaScript NoSQL document store designed to persist JSON data within web browsers or Node.js environments. It functions as an offline-first data store that caches information on a local device and synchronizes with a remote server when connectivity is available. The database implements the CouchDB API to ensure compatibility for bidirectional data replication. This allows for the synchronization of documents between a local client and remote CouchDB servers to maintain consistency across multiple devices. The project provides capabilities for local browser data storage and remot

Hadoop is a big data infrastructure suite and distributed data processing framework designed to store and process massive datasets across clusters of computers. It consists of a distributed storage system for managing large files across multiple nodes and a parallel computing engine for processing data across a distributed cluster. The framework implements a distributed file system to ensure fault tolerance and high throughput, paired with a programming model that processes large datasets in parallel. It manages the underlying hardware and software environment required for distributed big dat

This project is a collection of foundational machine learning algorithms and data science tools implemented in Python. It focuses on building the logic of these tools using basic programming primitives rather than relying on specialized libraries. The implementation covers several core domains, including a linear algebra library for matrix and vector operations, a statistical analysis toolkit for probability and hypothesis testing, and a framework for map-reduce distributed processing. It also includes implementations for natural language processing, graph theory for network analysis, and var

Apache Hive is a SQL-on-Hadoop data warehouse that enables querying and managing petabytes of data stored in distributed storage such as HDFS and cloud storage services. It provides a familiar SQL interface for batch analytics and reporting, supported by a core set of components including the HiveServer2 Thrift service for remote query execution, the Hive Metastore Service for central metadata management, the Hive ACID Transaction Engine for concurrent read-write operations, and the Hive LLAP Interactive Engine for low-latency analytical processing. The WebHCat REST API offers an HTTP interfac

HBase est un magasin NoSQL distribué à colonnes larges et un moteur de stockage de big data conçu pour les jeux de données creux. Il fonctionne comme une base de données colonnaire évolutive construite au-dessus du Hadoop Distributed File System pour fournir un accès en lecture et écriture en temps réel à des volumes massifs de données structurées et non structurées. Le système agit comme une passerelle de base de données multi-langage, offrant une connectivité via des appels de procédure distante (RPC) natifs, REST et des interfaces Thrift. Il se distingue par un modèle de coordination maître-esclave qui permet une mise à l'échelle horizontale et une tolérance aux pannes à travers un cluster. Le projet couvre un large ensemble de capacités, notamment le contrôle d'accès granulaire via des étiquettes de visibilité au niveau de la cellule, la compression de données enfichable et l'agrégation de données côté serveur. Il supporte également les workflows d'analyse de big data via l'intégration map-reduce et permet l'exécution de logique personnalisée côté serveur. La surveillance opérationnelle est fournie via le suivi des métriques système et l'exportation de métriques basée sur des plugins.

This is a collection of academic programming projects that accompany an operating systems textbook, designed to teach core OS concepts through hands-on implementation. The projects span the major subsystems of an operating system, including process scheduling, memory management, file systems, and concurrency, with students building components from scratch in a simulated environment. The projects are structured to cover the full range of OS internals, from low-level kernel development to user-space system programming. Students implement lottery-based CPU schedulers, dynamic heap memory allocat

This is an educational repository providing implementations and tutorials for deep learning, neural network architectures, and machine learning fundamentals. It serves as a reference for building multilayer perceptrons, convolutional networks, and recurrent networks using backpropagation and gradient descent. The project includes specialized frameworks for generative modeling via autoencoders and generative adversarial networks, as well as a toolkit for reinforcement learning that implements value-based, policy-based, and actor-critic methods. It also provides practical references for transfo