Awesome GitHub RepositoriesDataset Iterators

Ray is a distributed computing framework designed to scale Python and Java applications across clusters by abstracting task scheduling and resource management. It functions as a resource-aware execution engine that manages task dependencies, placement, and fault tolerance across networked compute nodes. At its core, the system provides a stateful actor model, allowing developers to define classes that run in dedicated processes to maintain and mutate internal state across remote method calls. The framework distinguishes itself through a robust cross-language interoperability layer, enabling f

This project is a deep learning framework designed for constructing, training, and deploying neural networks across diverse hardware environments. It functions as a high-performance tensor computation library that provides both imperative and symbolic programming interfaces, allowing developers to balance flexible, step-by-step model building with the efficiency of compiled computation graphs. The framework distinguishes itself through a hybrid execution engine that integrates declarative graph compilation with imperative runtime logic. It supports scalable, distributed training across multip

Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

Predis is a PHP library for connecting to and executing commands against Redis and Valkey data stores. It functions as a client for managing data integration, providing dedicated implementations for cluster sharding, pub/sub messaging, and Sentinel-based service discovery. The project distinguishes itself through specialized clients for executing server-side Lua scripts with automated hash caching and a cluster client that supports gossip protocols and key distribution. It also implements a Sentinel client to manage high availability and failover within replicated environments. The library c

DeepChem is an open-source Python framework for applying deep learning to molecular, chemical, and biological data, serving as a comprehensive toolkit for drug discovery and materials science. At its core, it provides a featurizer-pipeline abstraction that converts raw molecular data into numerical representations, including graph-based molecular structures, SMILES tokenization vocabularies, and disk-sharded dataset persistence for handling large-scale data that exceeds RAM capacity. The framework distinguishes itself through integrated molecular docking workflows that automate pocket detecti

Osmedeus is a security workflow orchestration engine that coordinates AI agents, shell commands, and scanning tools through declarative YAML pipelines. It functions as a distributed security scanner, a declarative workflow automator, and an AI agent framework for security, enabling automated multi-step security analysis with conditional branching, parallel execution, and distributed workers. The engine distinguishes itself through a hybrid runner model that executes workflow steps on the local host, inside Docker containers, or over SSH to remote machines, selected per step or module. It supp

chaiNNer is a GPU-accelerated AI image upscaling application that uses a visual node-based interface for constructing image processing pipelines. At its core, it provides a node-based visual programming environment where users connect processing nodes in a directed acyclic graph, with a graph execution scheduler that traverses the pipeline in topological order. The application includes an iterator-based batch processing system that automatically applies the same pipeline to multiple files, and a model format conversion pipeline that transforms neural network models between PyTorch, ONNX, and N

River ist ein Python-Framework für Online-Machine-Learning, das darauf ausgelegt ist, Modelle auf Streaming-Daten zu trainieren und zu evaluieren. Es ermöglicht inkrementelles Lernen durch die Aktualisierung von Modellparametern pro Beobachtung, wodurch das Speichern vollständiger Trainingsdatensätze im Arbeitsspeicher entfällt. Die Bibliothek zeichnet sich durch ein dediziertes System zur Erkennung von Concept Drift aus, das Änderungen in Datenverteilungen überwacht, um eine Modellanpassung auszulösen. Sie bietet zudem ein Framework für progressive Validierung, das den Echtzeit-Einsatz simuliert, indem Modelle an Stichproben getestet werden, bevor sie für das Training verwendet werden. Das System deckt ein breites Spektrum an Streaming-Funktionen ab, einschließlich Echtzeit-Feature-Engineering, Zeitreihenprognosen und Online-Anomalieerkennung. Es unterstützt unüberwachtes Lernen durch inkrementelles Clustering und Entscheidungsbäume sowie Ensemble-Aggregation und Bandit-Richtlinien für die Modellauswahl. Das Projekt enthält Dienstprogramme für das Streaming von Daten aus Quellen wie CSV-Dateien und APIs sowie Werkzeuge zur Berechnung laufender Statistiken und speichereffizienter Daten-Sketches.

Dieses Projekt bietet eine Observability-Datenpipeline, die darauf ausgelegt ist, Logs, Metriken und Traces aus verschiedenen Quellen zu sammeln, zu transformieren und in standardisierte Formate für die Analyse zu routen. Es arbeitet als Plugin-basierte Komponentenarchitektur, die modulare Receiver, Prozessoren und Exporter verwendet, um Telemetriedaten durch sequentielle Verarbeitungsketten zu bewegen. Das System nutzt ein schnittstellengetriebenes Komponentenmodell, das austauschbare Konnektoren und von der Community beigesteuerte Erweiterungen ermöglicht. Es zeichnet sich durch eine domänenspezifische Sprache für Telemetrie-Filterung, metadatenbasierte Ressourcenattribuierung für die Infrastrukturerkennung und dynamische Secret-Auflösung von externen Cloud-Managern aus. Der Collector deckt eine breite Palette an Funktionen ab, einschließlich Telemetrie-Ingestion von Cloud-Providern und Datenbanken, Datentransformation und Reaggregation sowie sicheren Export an Speicher-Backends von Drittanbietern. Er integriert Funktionen für das Traffic-Management wie Round-Robin-Routing und Nachrichtenpartitionierung sowie Sicherheitsprimitive für Identitäts- und Zugriffsmanagement via OAuth2 und OIDC. Das Projekt enthält ein Qualitätssicherungs-Framework für synthetische Datensimulation, End-to-End-Leistungstests und Datenintegritätsprüfung.

pe-sieve is a set of diagnostic tools for scanning Windows process memory to identify malicious implants, shellcode, and hooks. It functions as an in-memory implant detector, malware unpacker, and process callstack analyzer designed to locate and dump memory patches and injected code from running processes. The project identifies advanced evasion techniques, such as process hollowing and reflective injection, by verifying portable executable structures in memory. It distinguishes itself by analyzing process callstacks to detect anomalies and redirections and by reconstructing executable heade