Awesome GitHub RepositoriesDataset Iterators

Ray is a distributed computing framework designed to scale Python and Java applications across clusters by abstracting task scheduling and resource management. It functions as a resource-aware execution engine that manages task dependencies, placement, and fault tolerance across networked compute nodes. At its core, the system provides a stateful actor model, allowing developers to define classes that run in dedicated processes to maintain and mutate internal state across remote method calls. The framework distinguishes itself through a robust cross-language interoperability layer, enabling f

This project is a deep learning framework designed for constructing, training, and deploying neural networks across diverse hardware environments. It functions as a high-performance tensor computation library that provides both imperative and symbolic programming interfaces, allowing developers to balance flexible, step-by-step model building with the efficiency of compiled computation graphs. The framework distinguishes itself through a hybrid execution engine that integrates declarative graph compilation with imperative runtime logic. It supports scalable, distributed training across multip

Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

Predis is a PHP library for connecting to and executing commands against Redis and Valkey data stores. It functions as a client for managing data integration, providing dedicated implementations for cluster sharding, pub/sub messaging, and Sentinel-based service discovery. The project distinguishes itself through specialized clients for executing server-side Lua scripts with automated hash caching and a cluster client that supports gossip protocols and key distribution. It also implements a Sentinel client to manage high availability and failover within replicated environments. The library c

DeepChem is an open-source Python framework for applying deep learning to molecular, chemical, and biological data, serving as a comprehensive toolkit for drug discovery and materials science. At its core, it provides a featurizer-pipeline abstraction that converts raw molecular data into numerical representations, including graph-based molecular structures, SMILES tokenization vocabularies, and disk-sharded dataset persistence for handling large-scale data that exceeds RAM capacity. The framework distinguishes itself through integrated molecular docking workflows that automate pocket detecti

Osmedeus is a security workflow orchestration engine that coordinates AI agents, shell commands, and scanning tools through declarative YAML pipelines. It functions as a distributed security scanner, a declarative workflow automator, and an AI agent framework for security, enabling automated multi-step security analysis with conditional branching, parallel execution, and distributed workers. The engine distinguishes itself through a hybrid runner model that executes workflow steps on the local host, inside Docker containers, or over SSH to remote machines, selected per step or module. It supp

River 是一个用于在线机器学习的 Python 框架，旨在对流式数据进行模型训练和评估。它通过一次处理一个观测值来更新模型参数，从而实现增量学习，无需在内存中存储完整的训练数据集。 该库通过专门的概念漂移（Concept Drift）检测系统脱颖而出，该系统监控数据分布的变化以触发模型自适应。它还提供了一个渐进式验证框架，通过在训练前对样本进行测试来模拟实时部署。 该系统涵盖了广泛的流式处理功能，包括实时特征工程、时间序列预测和在线异常检测。它支持通过增量聚类和决策树进行无监督学习，以及用于模型选择的集成聚合和 Bandit 策略。 该项目包括从 CSV 文件和 API 等来源进行流式数据摄取的实用程序，以及用于计算运行统计信息和内存高效数据草图（Data Sketches）的工具。

chaiNNer is a GPU-accelerated AI image upscaling application that uses a visual node-based interface for constructing image processing pipelines. At its core, it provides a node-based visual programming environment where users connect processing nodes in a directed acyclic graph, with a graph execution scheduler that traverses the pipeline in topological order. The application includes an iterator-based batch processing system that automatically applies the same pipeline to multiple files, and a model format conversion pipeline that transforms neural network models between PyTorch, ONNX, and N

This project provides an observability data pipeline designed to collect, transform, and route logs, metrics, and traces from diverse sources into standardized formats for analysis. It operates as a plugin-based component architecture using modular receivers, processors, and exporters to move telemetry data through sequential processing chains. The system utilizes an interface-driven component model that allows for interchangeable connectors and community-contributed extensions. It distinguishes itself through a domain-specific language for telemetry filtering, metadata-based resource attribu

pe-sieve is a set of diagnostic tools for scanning Windows process memory to identify malicious implants, shellcode, and hooks. It functions as an in-memory implant detector, malware unpacker, and process callstack analyzer designed to locate and dump memory patches and injected code from running processes. The project identifies advanced evasion techniques, such as process hollowing and reflective injection, by verifying portable executable structures in memory. It distinguishes itself by analyzing process callstacks to detect anomalies and redirections and by reconstructing executable heade

Excel Streaming Reader is a Java library designed for processing massive spreadsheet files by reading data incrementally. It functions as a memory-efficient parser that prevents system memory exhaustion by avoiding the need to load entire documents into memory at once. The library utilizes an event-driven approach to translate low-level XML parsing events into high-level row and cell objects. By implementing streaming row iteration, it exposes data through a sequential cursor interface that maintains only a small, configurable subset of rows in memory during the extraction process. To suppor