Awesome GitHub RepositoriesPipeline Configurations

Kreuzberg is a document extraction engine that converts PDFs, Office files, images, and over 90 other formats into clean, structured text and metadata. It is built around a compiled Rust core that can be used as a native library, a command-line tool, a REST API server, or a WebAssembly module for browser-based processing. The system is designed to run entirely on self-hosted infrastructure, with no data leaving the user's environment. What distinguishes Kreuzberg is its breadth of integration surfaces and its pipeline architecture. It exposes extraction capabilities through native bindings fo

Proxyman is a cross-platform HTTP debugging proxy that captures, inspects, and modifies HTTP, HTTPS, and WebSocket traffic. It functions as a man-in-the-middle proxy, decrypting SSL/TLS traffic to allow real-time inspection and modification of encrypted requests and responses. The tool is designed for debugging web and mobile applications, with capabilities for API mocking and simulation, scriptable traffic modification, and team collaboration on network logs. What distinguishes Proxyman is its deep integration with mobile and cross-platform development workflows. It provides automated certif

frpc-desktop is a cross-platform desktop application that provides a graphical interface for managing FRP intranet penetration tunnels. It combines FRP binary downloading, visual configuration editing, tunnel lifecycle management, and shareable link generation into a single tool, allowing users to tunnel local network services to the public internet through a relay server without manually editing configuration files or managing binaries. The application wraps a web frontend inside an Electron shell for native window management across operating systems, and handles the full lifecycle of the FR

Data-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets to prepare them for training large language and vision models. It functions as a distributed data pipeline engine that runs processing jobs across Ray clusters, handling billions of samples with automatic operator fusion and adaptive parallelism. The framework provides a library of operators that leverage large language models for semantic extraction, filtering, and data synthesis within processing pipelines. The project distinguishes itself through a YAML-based data recipe sys

PaddleX is a PaddlePaddle-based framework for building, deploying, and fine-tuning AI model pipelines, with pre-built support for computer vision, OCR, document analysis, and time series tasks. It offers a toolkit of ready-to-use pipelines for image classification, object detection, segmentation, and pose estimation, alongside an end-to-end OCR document analysis pipeline that extracts text, tables, formulas, and layout information. The platform also includes a dedicated time series forecasting pipeline for analyzing historical data to detect anomalies, classify patterns, and predict future val

ZenML is an extensible machine learning orchestration framework designed to manage the end-to-end lifecycle of data pipelines and AI agent workflows. It functions as a durable orchestrator that executes machine learning tasks as directed acyclic graphs, ensuring that every step is containerized for consistent performance across local, cloud, and hybrid infrastructure. By decoupling pipeline code from underlying compute and storage backends, the platform allows developers to define infrastructure-agnostic stacks that remain portable across diverse environments. The project distinguishes itself

ZenML is an orchestration platform designed for building, deploying, and monitoring reproducible machine learning pipelines and agentic workflows. It provides a unified framework that manages the entire lifecycle of machine learning assets, from data processing and model training to the deployment of persistent inference services. By decoupling pipeline logic from underlying compute and storage, the platform enables teams to transition workflows seamlessly from local development environments to production-grade cloud infrastructure. The platform distinguishes itself through a service-oriented

sChandra is a document processing platform that converts images, PDFs, Word documents, spreadsheets, and other formats into structured output such as HTML, Markdown, or JSON while preserving layout. It can also extract specific data fields from invoices, contracts, or reports using user-defined JSON schemas, with citations back to source locations. The service supports form filling in PDF and image documents, document generation from Markdown, and extraction of tracked changes from Word files. The platform distinguishes itself with pipeline-based processing chains that combine multiple proces

Expose is a secure tunneling tool that creates public URLs for local development servers, making them accessible from the internet without complex network configuration. It forwards incoming HTTP requests and responses between a public relay server and a local machine, enabling external services to reach a development environment that would otherwise be isolated behind a firewall or VPN. The project distinguishes itself through a comprehensive set of webhook debugging and inspection capabilities. It captures all incoming webhook requests and displays them in a real-time dashboard or command-l

GPAC is an open-source multimedia framework built around a pluggable filter graph pipeline, where modular processing units called filters connect into a directed graph to handle media workflows. At its core, the framework centers all media packaging and manipulation on the ISO Base Media File Format (ISOBMFF), with specialized tools for reading, writing, fragmenting, and encrypting MP4 and related containers. It also provides a declarative scene graph composition system for describing interactive multimedia scenes using MPEG-4 BIFS, X3D, SVG, or VRML syntax, alongside a hardware-accelerated re

Le kit de données synthétiques est un framework intégré conçu pour générer, organiser et formater des jeux de données d'entraînement pour les modèles de langage. Il fournit un pipeline de bout en bout qui transforme les documents sources bruts en données structurées adaptées au fine-tuning, au raisonnement et à l'entraînement de modèles d'utilisation d'outils. Le framework se distingue par un moteur d'orchestration modulaire qui gère tout le cycle de vie de la préparation des données. Il prend en charge l'entrée multimodale en extrayant à la fois le contenu texte et image à partir de divers formats de fichiers, tout en utilisant un découpage conscient du contexte pour maintenir la cohérence sémantique. Le processus de génération est piloté par l'injection de prompts basée sur des modèles, et la sortie résultante est validée par un système d'évaluation automatisé qui utilise des modèles de langage comme juges pour garantir la qualité et la précision. Le projet couvre un large éventail de capacités de traitement de données, y compris l'analyse de documents, le filtrage de qualité automatisé et la sérialisation agnostique au schéma. Il prend en charge la création d'exemples d'entraînement divers, tels que des traces de raisonnement et des démonstrations d'utilisation d'outils, et exporte les jeux de données finaux dans des formats standardisés pour la compatibilité avec les frameworks d'entraînement de machine learning. Les utilisateurs gèrent le flux de travail de génération et les étapes du pipeline via des fichiers de configuration centralisés et des arguments en ligne de commande.