Awesome GitHub RepositoriesDeclarative Pipeline Construction

Pathway is a high-performance data processing framework designed for building unified batch and streaming pipelines. It functions as an orchestrator for complex data transformations, utilizing a differential dataflow engine to process updates incrementally. By treating static datasets and continuous event streams with identical logic, the platform ensures exactly-once processing semantics and consistent results across diverse data sources. The framework distinguishes itself through its specialized support for real-time artificial intelligence and retrieval-augmented generation. It features in

FFmpeg is a cross-platform multimedia framework designed for the recording, conversion, and streaming of audio and video content. It functions as a comprehensive toolkit that provides both a command-line utility for direct media manipulation and a collection of low-level libraries for integration into custom applications. At its core, the project utilizes a packet-based stream engine and a format-agnostic abstraction layer to handle diverse media standards, containers, and network protocols. The framework distinguishes itself through a modular, graph-based filter execution model that allows f

This tool is a command-line processor designed for querying, updating, and transforming structured data files. It functions as a versatile engine for manipulating YAML, JSON, TOML, and XML documents, allowing users to perform complex operations directly from the terminal. By utilizing a path-based expression language, it enables precise navigation and modification of data structures within configuration files and infrastructure-as-code workflows. What distinguishes this tool is its ability to perform in-place document mutations while preserving original formatting, comments, and metadata. It

Taskflow is a C++ task-parallel framework designed to build high-performance parallel workflows and complex dependency graphs. It provides a programming model that organizes computational work into directed acyclic graphs, enabling developers to manage concurrency, resource scheduling, and task dependencies across multi-core CPUs and GPU accelerators. The framework distinguishes itself through its ability to orchestrate heterogeneous systems, allowing for the integration of hardware-accelerated kernels and memory operations into unified execution pipelines. It supports dynamic runtime subflow

Benthos is a stream processing engine and data integration pipeline used for routing, transforming, and connecting data streams between diverse sources and sinks. It functions as event routing middleware and a change data capture tool, streaming real-time database modifications as discrete events for downstream processing. The system utilizes a declarative pipeline configuration, where data flow and processing logic are defined in a single static file. It features a specialized domain-specific language for mapping, filtering, and enriching data payloads, allowing for complex transformations w

node-fluent-ffmpeg هو غلاف Node.js لـ FFmpeg يوفر واجهة سلسة لتنفيذ أوامر الوسائط ومعالجة الملفات. يعمل كمدير عمليات يتعامل مع دورة حياة ثنائيات FFmpeg الخارجية، مما يتيح تحويل الوسائط برمجياً، وتوليد صور مصغرة للفيديو، واستخراج البيانات الوصفية عبر ffprobe. تتميز المكتبة بمنشئ أوامر يترجم استدعاءات أساليب JavaScript إلى وسيطات سطر أوامر. تتميز بمراقبة التقدم القائمة على الأحداث لتتبع الإطارات المعالجة والإنتاجية، بالإضافة إلى القدرة على توجيه بيانات الوسائط المعالجة مباشرة إلى تدفقات قابلة للكتابة للتعامل في الوقت الفعلي. يغطي المشروع قدرات واسعة لمعالجة الوسائط، بما في ذلك إعداد التشفير لخصائص الصوت والفيديو، وتعريفات مخططات التصفية المعقدة للتأثيرات المرئية والصوتية، وإدارة المدخلات لربط مصادر متعددة. يتضمن أيضاً أدوات لفحص حاويات الوسائط والتدفقات لاسترجاع البيانات الوصفية التقنية.

This repository is a comprehensive educational program and deep learning framework designed to teach practical deep learning using PyTorch through notebooks and code examples. It serves as a high-level library for building, training, and deploying neural networks, acting as a model training orchestrator that coordinates PyTorch models, optimizers, and loss functions. The project provides specialized toolkits for computer vision, natural language processing, and tabular data preprocessing. It distinguishes itself through advanced training controls such as discriminative learning rates, a two-w

docetl is an AI-powered document ETL tool and map-reduce orchestrator designed to transform large collections of unstructured documents into structured, queryable tables using language models. It provides a declarative pipeline framework for extracting, cleaning, and transforming data from sources such as PDFs and text files into predefined schemas. The project distinguishes itself through a semantic data integration suite that enables joining datasets and resolving duplicate entities based on embedding-based similarity. It includes an interactive prompt playground for developing and optimizi

This is a structured deep learning curriculum for programmers, delivered as a collection of Jupyter notebooks. It teaches the fundamentals of training neural networks for computer vision, natural language processing, tabular data analysis, and collaborative filtering using PyTorch and the fastai library. The course is designed to be hands-on, guiding learners from building a training loop from scratch to fine-tuning pretrained models for a variety of practical tasks. The curriculum distinguishes itself by covering the full lifecycle of a deep learning project, from data preparation and augmen

Dag-factory هو إطار عمل لبناء وإدارة خطوط أنابيب بيانات Apache Airflow من خلال ملفات تكوين تعريفية. من خلال استبدال الكود الإجرائي اليدوي بتعريفات YAML منظمة، فإنه يتيح الإنشاء البرمجي لهياكل سير العمل المعقدة، وتبعيات المهام، وجداول التنفيذ. يتميز المشروع بربط مفاتيح التكوين مباشرة بمنشئات فئات Python والمشغلين، مما يسمح بالإنشاء الديناميكي للكائنات والمنطق المخصص. كما يدعم توريث التكوين الهرمي لتوحيد الإعدادات عبر البيئات ويوفر آليات لحقن مواصفات حاويات Kubernetes مباشرة في تعريفات المهام لضمان تنفيذ معزول وقابل للتوسع. يغطي إطار العمل دورة حياة خط الأنابيب بالكامل، بما في ذلك اكتشاف الملفات الآلي، والتعيين الديناميكي على مستوى المهمة للمعالجة المتوازية، وإرفاق البيانات الوصفية لتكامل النظام الخارجي. كما يتضمن أدوات سطر أوامر للتحقق من التكوينات، وتشغيل التنفيذ، وإدارة ترحيلات البيئة.