apache/nifi
Nifi
Apache NiFi is a flow-based programming platform that enables the visual design, monitoring, and management of data pipelines. At its core, it provides a web-based visual dataflow designer where users build directed graphs of processors to route, transform, and mediate data movement between any source and destination without writing custom code. The system records fine-grained data provenance for every data item from ingestion to delivery, supporting audit, debugging, and replay of data lineage.
The platform distinguishes itself through a zero-master cluster architecture that distributes processing horizontally with back pressure-driven flow control, automatically slowing or stopping upstream processors when downstream capacity is exceeded to prevent data loss. It supports live configuration reloading, allowing changes to dataflow graph topology and processor properties at runtime without stopping the system or losing data. Content-based routing directs data to different destinations by evaluating attributes or content against configurable flow-based rules, while priority-queued data buffering controls processing order within the dataflow.
NiFi provides real-time monitoring of data movement, queue sizes, and processor status through its browser interface, with the ability to configure delivery guarantees such as loss-tolerant versus guaranteed delivery and low latency versus high throughput. The system secures data in transit and at rest using TLS, SSH, and HTTPS, enforcing multi-tenant authorization and policy management. The project is released through a formal process with signed artifacts and provides guidance for code contributions, commit signing, and licensing compliance.
Features
- Data Pipeline Orchestration - Visually builds, monitors, and manages data flows between any source and destination with real-time feedback.
- Data Pipeline Orchestrators - Provides a visual flow-based platform for orchestrating data movement and transformation between any source and destination.
- Processor Graph Dataflow Models - Provides a flow-based programming platform that connects processors into directed graphs for data movement without custom code.
- Back-pressure Handling - Slows or stops upstream processors when downstream capacity is exceeded to prevent data loss.
- Dataflow Back-Pressure Controllers - Provides automatic back-pressure-driven flow control that slows upstream processors when downstream capacity is exceeded.
- In-Transit Schema Transformations - Applies processors to modify, enrich, or restructure data as it moves through the flow.
- Data Integration Pipelines - Transfers data from any source to any destination using a flow-based programming model with configurable processors.
- Data Pipeline Configurations - Defines the flow of data between sources and destinations by connecting processors and configuring their properties.
- Horizontal Scaling - Distributes processing across a cluster using a zero-master model to handle increasing data volumes.
- Content-Based Data Routers - Directs data to different destinations based on its attributes or content using configurable flow-based rules.
- Dataflow Pipeline Designers - Provides a web-based visual dataflow designer for building, monitoring, and modifying data pipelines through a browser interface.
- Live Configuration Reloading - Supports live configuration reloading, allowing changes to dataflow graph topology and processor properties at runtime without stopping the system.
- Zero-Master Cluster Topologies - Implements a zero-master cluster architecture that distributes processing horizontally with no single point of control.
- Content-Based Data Routers - Directs data to different destinations by evaluating attributes or content against configurable flow-based rules.
- Live Dataflow Graph Modifiers - Changes the dataflow graph at runtime without stopping the system or losing data.
- Visual Dataflow Graph Designers - Designs a directed graph of data routing, transformation, and system mediation logic through a web-based user interface.
- Processor Graph Platforms - Connects processors in a directed graph to route, transform, and mediate data movement without coding.
- Back Pressure Data Routers - Distributes processing across a cluster with back pressure and priority queuing for horizontal scaling.
- Data Provenance Recorders - Records full lineage of every data item from ingestion to delivery for audit, debugging, and replay.
- Data Lineage Recorders - Records full lineage and provenance for every data item from ingestion to delivery for audit and debugging.
- Live Pipeline Configuration Modifiers - Changes the configuration of a running data pipeline without stopping or restarting the system.
- Real-Time Monitoring Dashboards - Monitors the status, feedback, and performance of running dataflows through a real-time web-based dashboard.
- Dataflow Movement Trackers - Tracks the movement and transformation of data through the flow with a web-based monitoring interface.
- Dataflow Performance Dashboards - Watches data movement, queue sizes, and processor status live through a web dashboard with provenance tracking.
- Dataflow Monitors - Provides real-time visibility into data movement, queue sizes, and processor status through a web interface.
- Visual Data Pipeline Builders - Provides a browser-based interface to build, control, monitor, and receive feedback on data pipelines.
- Dataflow Pipeline Designers - Designs and modifies data pipelines through a browser interface with live configuration changes.
- Data Pipeline Flow Designers - Designs data pipelines through a web interface to build, control, monitor, and receive feedback on their operation.
- Visual Pipeline Builders - Designs and modifies data flows between any source and destination through a browser-based interface with live configuration changes.
- Data Attribute Routing Engines - Ships a content-based routing engine that directs data to different destinations by evaluating attributes or content against flow-based rules.
- Encrypted Transfer Orchestrators - Encrypts data in transit with TLS and SSH while enforcing multi-tenant authorization and policy management.
- Data Encryption - Encrypts data in transit using TLS and SSH and enforces multi-tenant authorization and policy management.
- Multi-Tenant Transfer Security - Encrypts communication with TLS and SSH, authenticates users via HTTPS, and enforces multi-tenant authorization policies.
- Delivery Guarantees - Ships configurable quality-of-service settings for loss-tolerant versus guaranteed delivery and latency versus throughput.
- Dataflow - Implements priority-queued data buffering that controls processing order within the dataflow based on configurable priorities.
- Dataflow Priority Queue Managers - Queues data with priority-based ordering to control processing order within the dataflow.
- Data Integration - Automated data integration tool featuring a visual interface for data flow.
- Data Pipelines - Manages directed graphs for data routing and transformation.
