Open-source alternatives to Flyte

Inngest is a durable execution framework and event-driven automation engine designed to orchestrate background workflows. It enables developers to build resilient, stateful processes by memoizing function steps, ensuring that long-running tasks can automatically resume from the last successful operation after failures, timeouts, or infrastructure restarts. The platform distinguishes itself through its event-driven architecture, which uses a schema-validated bus to trigger functions and coordinate complex, multi-step logic. It employs an onion-model middleware approach for cross-cutting concer

SynapseML is an Apache Spark machine learning library designed for building and scaling machine learning workflows and data pipelines across distributed clusters. It serves as a distributed machine learning pipeline framework and a distributed inference engine for executing hardware-accelerated predictions and deep learning tasks on large-scale datasets. The project functions as a cloud AI integration layer, allowing users to apply pretrained artificial intelligence services for text, vision, and speech within distributed pipelines. It also includes a dedicated suite of tools for distributed

Serve is a multimodal AI orchestrator and inference server designed for deploying and scaling machine learning models as cloud-native services. It functions as a containerized workflow engine and distributed service mesh that routes multimodal data through connected execution units. The framework provides specialized capabilities for large language models, including a token streaming gateway that delivers generated text incrementally to reduce perceived latency. It distinguishes itself by enabling the chaining of executors into complex data processing pipelines and the orchestration of these

Exegol is an offensive security platform and containerized tooling orchestrator designed to deploy and manage isolated security operations environments. It functions as a workspace manager that provisions pre-configured security images and toolkits within Docker containers to protect host systems from malicious payloads. The platform distinguishes itself by integrating AI security workflow orchestration, allowing AI assistants to discover and trigger security tools through a standardized communication protocol. It further provides remote desktop gateway capabilities, enabling GUI access via X

Superduper is an AI agent development kit and LLM application framework designed to build autonomous agents and data-driven applications. It functions as a RAG orchestration platform and vector search infrastructure, coordinating AI models with database storage to perform multi-step computations and actions using persisted data states. The project distinguishes itself by providing a database-integrated machine learning pipeline that executes training and inference tasks directly on data hosted within SQL and NoSQL databases. It allows for the deployment of self-hosted AI infrastructure on pri

Pipeline is a Kubernetes native CI/CD framework and cloud native pipeline orchestrator. It functions as a custom resource controller that translates declarative pipeline definitions into coordinated pod executions and managed workloads. The system acts as a containerized task runner, allowing for the execution of standalone build steps and reusable tasks that process specific inputs to produce defined outputs. It enables the orchestration of complex workflows by running a sequence of independent containers as modular components within a cloud environment. The platform covers automated softwa

AISystem is a comprehensive AI full-stack infrastructure project covering the entire pipeline from AI chip architecture to high-level training frameworks. It encompasses the development of AI compiler frameworks, inference engines, and distributed training orchestrators designed to coordinate workloads across a heterogeneous compute stack of CPUs, GPUs, and NPUs. The project focuses on the deep integration of software and hardware, employing software-hardware co-design to align tensor layouts with physical memory structures. It provides specialized capabilities for accelerating Transformer mo

This project is a comprehensive toolkit designed for the full lifecycle management of large language and multimodal models. It functions as a unified orchestrator that handles the entire development process, ranging from dataset preparation and supervised fine-tuning to advanced reinforcement learning alignment and production-ready inference deployment. The platform distinguishes itself through a specialized reinforcement learning library that supports complex optimization algorithms, including group relative policy optimization and leave-one-out techniques, to improve model instruction-follo

Azkaban is a distributed workflow manager and DAG-based job orchestrator designed as an enterprise batch processor. It serves as a Java-based workflow engine that schedules and executes complex job sequences across a cluster of executor servers, with specific functionality for managing big data workloads on Hadoop clusters. The system distinguishes itself through a distributed executor model that coordinates state via a shared database to ensure high availability. It employs a plugin-based architecture that allows for custom job types and system functionality extensions, including the ability

This project is a structured educational program and comprehensive training curriculum designed to teach the end-to-end lifecycle of machine learning models. It serves as a resource for engineers to master the transition of data science projects from development into reliable, production-ready systems. The curriculum focuses on the practical application of engineering best practices, emphasizing the orchestration of complex data processing and training sequences. It provides instruction on building repeatable workflows, managing experiment metadata, and implementing infrastructure automation

This project is a learning curriculum and programming guide for Apache Spark, providing a structured set of educational resources and practical code examples for mastering distributed data processing. It serves as a course for building scalable data workflows and big data engineering pipelines. The repository provides practical source code and project layouts that demonstrate how to connect external data stores, process streaming data, and organize code for distributed environments. It includes implementation examples for scaling machine learning algorithms across clusters to handle large tra

Deep Java Library is a Java deep learning framework and JVM model inference engine. It provides a high-level API for building and deploying deep learning models within the Java ecosystem, acting as a cross-platform runtime for executing models across CPUs, GPUs, and mobile devices. The library is engine-agnostic, allowing users to switch between different deep learning engines such as PyTorch, TensorFlow, and MXNet while maintaining a single unified API. This enables the deployment of the same model across different backends without changing the application code. The framework supports the f

Flyte is a Kubernetes-based machine learning orchestrator and containerized pipeline manager designed for coordinating AI workflows and data pipelines. It functions as an engine for defining and executing resilient pipelines, utilizing a data lineage tracker to maintain immutable execution states and ensure reproducible outputs. The platform distinguishes itself by packaging individual tasks into separate containers to ensure dependency isolation and environment consistency. It provides specialized capabilities for machine learning, including the transformation of trained models into scalable

Nextflow is a dataflow workflow engine and distributed computing framework used to build and execute data-intensive pipelines. It serves as a scientific workflow language that allows users to define reproducible data processing sequences, supporting any scripting language through shebang declarations. The system functions as a containerized pipeline orchestrator, utilizing container technologies to ensure software dependencies remain consistent across different environments. It decouples workflow logic from the underlying infrastructure, enabling the same pipeline to run on local machines, cl

Cadence is a distributed workflow orchestration engine designed to execute long-running, asynchronous business logic with built-in durability and resilience across distributed systems. It functions as a stateful process manager that ensures processes resume from their last known state following system crashes or network outages. The platform utilizes a distributed task queue to manage work across independent worker nodes and supports persistence via SQL or Cassandra backend storage. It includes a workflow visualization dashboard for inspecting execution histories and state traces, alongside a

Windmill is an internal developer platform and workflow orchestration engine designed to automate complex business processes and data pipelines. It functions as a distributed serverless runner that executes multi-language scripts within isolated, containerized environments, allowing teams to chain discrete tasks into directed acyclic graphs. The platform distinguishes itself through a Git-centric approach to infrastructure, where system state and workflow definitions are synchronized directly from version control. It features a metadata-driven input system that automatically generates user in

The Fireworks Workflow Management Repo.

Genkit is an LLM application framework and generative AI developer toolkit designed for building production AI applications. It serves as an AI workflow orchestrator that coordinates model calls and agentic tool usage through type-safe execution flows. The project provides a unified model interface and plugin architecture to standardize access to diverse large language models, vector stores, and telemetry backends. It distinguishes itself with a dedicated observability suite for tracing execution steps and a developer toolkit for prompting, debugging, and evaluating AI logic via a local inter

This project is a Java-based framework integration that provides an AI agent runtime, a graph-based AI workflow engine, and an LLM orchestration framework for Spring applications. It enables the development of stateful autonomous agents and the implementation of retrieval-augmented generation systems using document processing and vector databases. The framework distinguishes itself through a graph-based workflow runtime for designing complex AI pipelines with conditional routing and persistent state. It supports multi-agent orchestration via service-discovery coordination and provides human-i

CL4R1T4S is a framework designed to orchestrate generative AI workflows and optimize language model outputs. It functions as a centralized utility for managing, versioning, and deploying structured system prompts and behavioral parameters to ensure consistent performance across complex tasks. The project distinguishes itself by implementing a structured pipeline that wraps model interactions to enforce behavioral constraints and sanitize inputs. This orchestration layer incorporates heuristic-based validation and stateful context management to maintain coherence and quality throughout multi-s

ComfyUI is a modular generative AI workflow orchestrator and node-based GUI for designing and executing complex diffusion model pipelines. It functions as both a visual interface for building generative logic graphs and a programmable backend API that exposes diffusion model operations for external integration. The system distinguishes itself through a graph-based execution model that supports differential workflow execution, re-running only modified nodes to reduce computation. It features dynamic model offloading to manage memory between system RAM and GPU VRAM and utilizes metadata-embedde

This repository contains the comprehensive documentation for a code editor focused on AI-assisted software development and remote development workflows. It covers the implementation of AI agents and language models used for autonomous code generation, large-scale refactoring, and task iteration. The project is distinguished by its deep integration of autonomous AI agents capable of web navigation, application logic validation, and orchestrating multi-step development processes. It provides specialized frameworks for tailoring AI behavior through custom instructions, model context protocols, a

Agent-OS is an LLM multi-agent orchestration framework and AI software development lifecycle tool designed to coordinate specialized agents through shared workspaces and structured task lists. It functions as an agentic application bootstrapper and technical specification engine, providing the infrastructure to guide the process from product requirements to automated coding and deployment. The system distinguishes itself through spec-driven development, using detailed technical specifications and layered context injection to ensure generated code aligns with project standards. It employs a ma

This project provides an advanced English curriculum and a set of instructional guides designed to help non-native speakers move from intermediate to advanced proficiency. It functions as a guide for AI-powered language training, utilizing structured workflows and prompt engineering with large language models to facilitate self-directed study. The system implements AI workflow orchestration, chaining different artificial intelligence models into feedback loops to automate linguistic exercises and corrections. This approach combines multiple AI specializations to coordinate training across lis

Ottomator-agents is a framework for building and deploying autonomous AI agents using structured workflow files and source code. It serves as a declarative deployment tool and workflow orchestrator that translates static configuration files into executable sequences of AI agent tasks and logic flows. The system utilizes manifest-driven instantiation and template-driven deployment to create functional agent identities by populating source code templates with user-specified parameters. It incorporates a modular skill system that equips agents with discrete, reusable source code units and toolse

Eino is an AI agent development kit and LLM application framework designed for building autonomous agents and orchestrating complex language model workflows. It serves as a multi-agent orchestration engine and workflow orchestrator, providing a graph-based execution model to route data between models, tools, and retrievers. The framework distinguishes itself through a robust set of multi-agent coordination patterns, including supervisor-led management, sequential flows, and autonomous reasoning loops like ReAct. It features advanced agent execution controls such as active turn preemption, che

This project is an AI-powered IDE extension and LLM coding assistant that provides a conversational interface for generating, refactoring, and debugging code. It functions as an AI agent framework and a Model Context Protocol client, connecting AI models to external data sources and tools to automate complex development tasks. The system is distinguished by its use of autonomous AI agents capable of multi-step task execution, including the ability to read files, modify code, and run terminal commands iteratively. It supports recursive agent orchestration through subagent delegation and employ