Open-source alternatives to Bottledwater Pg

This project is an AWS pandas integration library and data pipeline framework designed to simplify the movement and transformation of data between local memory and AWS storage and analytics services. It functions as a cloud data lake toolkit and storage file manager, allowing users to read, write, and transform structured data across various cloud environments. The library distinguishes itself as a distributed compute orchestrator capable of managing clusters in environments such as EMR to process datasets that exceed the memory limits of a single machine. It also provides specialized capabil

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

ingestr is a command-line tool for copying and syncing data between different database engines and third-party platforms without writing custom code. It functions as an ETL pipeline utility that extracts data from diverse sources and loads it into destinations. The tool features a schema-agnostic data loader that maps source fields to destination columns dynamically, removing the need for predefined static table definitions. It also operates as an incremental data synchronizer, updating destination tables by appending new records or merging changes to maintain current datasets. The system pr

Data ingestion engine — Singer taps to Parquet in cloud buckets

Kafkacat is a suite of command-line utilities for interacting with Apache Kafka clusters. It provides a non-JVM binary for producing and consuming messages, inspecting cluster metadata, and debugging the Kafka protocol via the terminal. The tool functions as a producer and consumer capable of pushing data from files or standard input and reading messages from specific topics and partitions. It includes a metadata inspector to retrieve cluster state and partition configurations in plain text or JSON, as well as a protocol debugger for inspecting message offsets, timestamps, and binary payloads

The Apache Kafka C/C++ library

Scribe is a distributed log aggregation system designed to collect and route real-time log data from numerous servers to centralized storage or analysis tools. It functions as a log data pipeline and scalable collector that gathers streaming data and writes it to local disks or remote endpoints. The system employs a log routing server model that organizes incoming streams into specific buckets based on predefined configuration mappings. It supports multi-hop log forwarding, allowing data to be routed through a chain of intermediate servers to centralize logs from diverse network segments. Re

Live import all your Google Sheets to your data warehouse

Build platforms that flexibly mix SQL, batch, and stream processing paradigms

Kreuzberg is a document extraction engine that converts PDFs, Office files, images, and over 90 other formats into clean, structured text and metadata. It is built around a compiled Rust core that can be used as a native library, a command-line tool, a REST API server, or a WebAssembly module for browser-based processing. The system is designed to run entirely on self-hosted infrastructure, with no data leaving the user's environment. What distinguishes Kreuzberg is its breadth of integration surfaces and its pipeline architecture. It exposes extraction capabilities through native bindings fo

Kroxylicious, the snappy open source proxy for Apache Kafka®

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

DocId set compression and set operation library

Hadoop log aggregator and dashboard

Conflict-free merge for DataFrames, JSON, ML models & distributed agents — powered by CRDTs. The first merge library where every operation is mathematically guaranteed to converge.

DEPRECATED: Data collection and processing made easy.

PDF extraction that checks its own work. #2 reading order accuracy — zero AI, zero GPU, zero cost.

Netflix's distributed Data Pipeline

Secor is a service implementing Kafka log persistence

High performant ETL engine written in TypeScript

Rudder Server is a customer data platform and event routing pipeline designed to collect, transform, and route customer event data from various sources to data warehouses and business tools. It functions as a customer identity resolver, linking identifiers from multiple sources to build a unified identity graph and comprehensive behavioral customer profiles. The system differentiates itself through reverse ETL capabilities, which push processed customer segments and audiences from data warehouses back into operational third-party applications. It also provides a containerized data plane for K

A probabilistic data structure service and storage

Node.js client for Apache Kafka 0.8 and later.

Hadoop Data Integration with various databases, ftp servers, salesforce. Incremental update, dedup, append, merge your data on Hadoop.