Awesome GitHub RepositoriesMapReduce Processing Engines

Redisson is a Java client library for Redis and Valkey that provides a distributed data structure library, a distributed lock manager, and a distributed MapReduce framework. It enables application instances in a cluster to share state through thread-safe collections and objects. The project implements a JCache compliant caching layer for standardized data storage and retrieval. It also functions as a probabilistic data store, providing memory-efficient structures such as Bloom filters and HyperLogLog for high-volume data membership testing. The library covers distributed state management usi

This project serves as a comprehensive technical reference for the architecture and design of data-intensive applications. It provides a structured analysis of the fundamental principles required to build reliable, scalable, and maintainable software systems, covering the core trade-offs inherent in modern data infrastructure. The repository explores the mechanics of distributed data management, including strategies for replication, partitioning, and achieving consensus across multiple nodes. It details the design of storage engines, indexing techniques, and transaction management models, whi

PouchDB is a JavaScript NoSQL document store designed to persist JSON data within web browsers or Node.js environments. It functions as an offline-first data store that caches information on a local device and synchronizes with a remote server when connectivity is available. The database implements the CouchDB API to ensure compatibility for bidirectional data replication. This allows for the synchronization of documents between a local client and remote CouchDB servers to maintain consistency across multiple devices. The project provides capabilities for local browser data storage and remot

Hadoop is a big data infrastructure suite and distributed data processing framework designed to store and process massive datasets across clusters of computers. It consists of a distributed storage system for managing large files across multiple nodes and a parallel computing engine for processing data across a distributed cluster. The framework implements a distributed file system to ensure fault tolerance and high throughput, paired with a programming model that processes large datasets in parallel. It manages the underlying hardware and software environment required for distributed big dat

This project is a collection of foundational machine learning algorithms and data science tools implemented in Python. It focuses on building the logic of these tools using basic programming primitives rather than relying on specialized libraries. The implementation covers several core domains, including a linear algebra library for matrix and vector operations, a statistical analysis toolkit for probability and hypothesis testing, and a framework for map-reduce distributed processing. It also includes implementations for natural language processing, graph theory for network analysis, and var

Apache Hive is a SQL-on-Hadoop data warehouse that enables querying and managing petabytes of data stored in distributed storage such as HDFS and cloud storage services. It provides a familiar SQL interface for batch analytics and reporting, supported by a core set of components including the HiveServer2 Thrift service for remote query execution, the Hive Metastore Service for central metadata management, the Hive ACID Transaction Engine for concurrent read-write operations, and the Hive LLAP Interactive Engine for low-latency analytical processing. The WebHCat REST API offers an HTTP interfac

HBase 是一个分布式、宽列 NoSQL 存储和大数据存储引擎，专为稀疏数据集设计。它作为一个可扩展的列式数据库，构建在 Hadoop 分布式文件系统之上，提供对海量结构化和非结构化数据的实时读写访问。 该系统充当跨语言数据库网关，通过原生远程过程调用、REST 和 Thrift 接口提供连接。它通过主从协调模型脱颖而出，该模型实现了跨集群的水平扩展和容错能力。 该项目涵盖了广泛的功能，包括通过单元级可见性标签实现的细粒度访问控制、可插拔数据压缩和服务器端数据聚合。它还通过 MapReduce 集成支持大数据分析工作流，并允许执行自定义的服务器端逻辑。 操作监控通过系统指标跟踪和基于插件的指标导出提供。

This is a collection of academic programming projects that accompany an operating systems textbook, designed to teach core OS concepts through hands-on implementation. The projects span the major subsystems of an operating system, including process scheduling, memory management, file systems, and concurrency, with students building components from scratch in a simulated environment. The projects are structured to cover the full range of OS internals, from low-level kernel development to user-space system programming. Students implement lottery-based CPU schedulers, dynamic heap memory allocat

这是一个教育性仓库，提供深度学习、神经网络架构和机器学习基础的实现与教程。它作为构建多层感知器、卷积网络和循环网络的参考，涵盖了反向传播和梯度下降等核心概念。 该项目包括用于通过自动编码器和生成对抗网络（GAN）进行生成式建模的专用框架，以及一个实现基于价值、基于策略和 Actor-Critic 方法的强化学习工具包。它还提供了 Transformer 和 BERT 架构的实用参考，重点关注自然语言处理和视觉数据任务中的注意力机制。 该仓库涵盖了广泛的能力，包括计算机视觉处理、序列建模和对抗鲁棒性分析。它还提供了分布式机器学习指南，详细介绍了使用 MapReduce、参数服务器和联邦学习在多个节点上扩展训练的策略。 该项目为传统机器学习算法提供了基础支持，具体涵盖回归、分类和聚类。