Awesome GitHub RepositoriesPython Data Pipeline Frameworks

This project is a Python workflow orchestration platform and programmatic data pipeline engine used to author, schedule, and monitor complex data pipelines. It functions as a directed acyclic graph manager and scheduler, allowing users to define data movement and transformation tasks as code to ensure precise execution order and maintainability. The platform distinguishes itself by treating workflows as code, enabling pipelines to be versioned and tested through a standard programming language. It utilizes a system of extensible operators to encapsulate integration logic and employs a templat

Luigi is a Python framework designed for building and managing complex batch data pipelines. It functions as a workflow orchestration engine that organizes tasks into directed acyclic graphs, ensuring that jobs execute in the correct logical order based on their dependencies. By utilizing a centralized scheduler, the system coordinates task execution across distributed environments, tracks global workflow state, and prevents redundant processing by verifying the existence of output targets before triggering any work. The project distinguishes itself through a robust state-tracking mechanism t

Grist is a relational spreadsheet platform that combines the flexibility of a spreadsheet with the power of a relational database. At its core, it manages structured data across multiple linked tables, using a relational database engine to organize information while providing a familiar grid interface. The platform supports Python-based formulas for complex calculations and data transformations, with automatic recalculation when referenced cells change. The system is designed for self-hosted deployment, storing data in either portable SQLite files or enterprise-grade PostgreSQL databases. It

Kedro is a data science pipeline framework and orchestration tool designed to build reproducible and modular data engineering workflows. It functions as an MLOps project template and Python data workflow tool that enforces software engineering best practices to move projects from prototype to production. The system distinguishes itself through a centralized data catalog manager that abstracts data access and versioning across various file formats and cloud storage systems. It further separates processing logic from data access via a lazy-loading data registry and provides a standardized proje

Mage AI 是一个基于 Python 的数据流水线编排器和自托管数据集成开发环境。它旨在通过基于块的流水线设计和交互式笔记本界面来构建、调度和监控数据工作流。 该平台通过集成生成式 AI 功能脱颖而出，允许用户通过 API 连接大语言模型提供商，将人工智能纳入自动化数据流中。它还作为一个 Apache Spark 数据处理器，管理高性能分析和大规模数据处理所需的内核和基础设施。 该系统涵盖了广泛的数据工程功能，包括 ETL 工作流自动化、dbt 模型管理和数据流发现。它提供了通过 Git 进行版本控制集成、容器化部署以及基于角色的访问控制的工具，以管理跨开发和生产环境的流水线。监控通过系统性能遥测和流水线执行调试进行处理。

Tensorpack 是一个高级 TensorFlow 神经网络框架和研究库，专为构建和训练深度学习模型而设计。它提供了一系列可复现的神经网络架构，用于计算机视觉、生成任务、强化学习和自然语言处理。 该项目通过一个专门的深度学习数据流水线脱颖而出，该流水线使用纯 Python 进行并行数据加载和流式传输。它包括一个用于通过数据并行策略分发工作负载的多 GPU 训练编排器，以及一个用于可视化模型显著性和激活图的专用可解释性工具包。 该框架涵盖了广泛的功能，包括用于目标检测和语义分割的计算机视觉流水线、用于语音和文本的序列建模，以及强化学习代理开发。它还提供用于权重量化和低位宽训练的模型优化工具，以及用于复现学术研究论文和转换遗留 Caffe 模型权重的实用程序。

Toolz is a Python library that implements functional programming utilities for iterable transformation, dictionary manipulation, function composition, and lazy evaluation. It provides a set of pure functions designed to work with Python's built-in data structures, enabling concise and composable data processing workflows. What distinguishes toolz is its support for curried partial application, allowing functions to be incrementally applied and reused. It includes dictionary-centric operations that handle nested structures, and offers iterable chain transformers that combine mapping, filtering

该项目是一个 AWS pandas 集成库和数据流水线框架，旨在简化本地内存与 AWS 存储及分析服务之间的数据移动和转换。它作为一个云数据湖工具包和存储文件管理器，允许用户在各种云环境中读取、写入和转换结构化数据。 该库作为分布式计算编排器脱颖而出，能够在 EMR 等环境中管理集群，以处理超出单机内存限制的数据集。它还提供用于管理向量索引和在云存储桶内执行相似度搜索的专门功能。 其更广泛的功能面涵盖了针对 DynamoDB、RDS 和 Timestream 等服务的云数据库 ETL，以及通过 AWS Glue 进行的云数据目录管理。它支持通过 Athena 和 Redshift 进行无服务器数据分析，并提供用于管理 S3 对象、在 OpenSearch 中索引文档以及分析 CloudWatch 日志的实用程序。

This project is a collection of interactive Jupyter notebooks designed to teach machine learning and deep learning fundamentals through hands-on coding exercises. It provides a structured curriculum that guides users through the end-to-end data science lifecycle, covering everything from initial data preprocessing to final model evaluation. The repository distinguishes itself by bridging theoretical data science concepts with practical implementation using standard industry libraries. It features a series of tutorials that demonstrate how to build and train predictive models and complex neura