Open-source alternatives to Datasharing

This project provides educational materials and courseware focused on the theoretical and practical foundations of distributed systems design. It serves as a comprehensive curriculum covering the disciplines of consensus, data consistency, reliability engineering, and scalability. The instructional content focuses on achieving cluster agreement through consensus algorithms and managing system-wide state via coordination frameworks. It includes a dedicated guide to data theory, exploring replication strategies, consistency models, and data convergence. The courseware covers a broad capability

This project is a collection of interactive Python notebooks and educational resources designed for mastering data science, machine learning, and numerical computing. It provides a series of practical guides and tutorials covering deep learning, big data processing, and statistical analysis. The repository features specialized instructional suites for implementing classical machine learning algorithms, building deep learning model architectures, and managing AWS cloud infrastructure. It includes dedicated notebooks for data visualization and numerical computing exercises. The project covers

hledger is a plain text accounting tool and double-entry ledger manager that stores financial transactions in human-readable text files. It functions as a financial reporting engine for generating balance sheets and income statements, and as a multi-currency investment tracker for managing commodity lots and capital gains. The project distinguishes itself by providing multi-interface data access, allowing users to interact with their financial data via a command line interface, a terminal user interface, and a web server. It features a market-price valuation system to calculate the current va

dplyr is an R data manipulation library that provides a grammar for transforming tabular data frames. It functions as an in-memory data frame processor and a relational data algebra tool, using a consistent set of verbs to filter, select, and summarize data. The project includes a SQL translation engine that converts high-level data manipulation expressions into optimized queries. This allows users to perform transformations directly on remote relational databases and cloud storage without pulling data locally. The library covers a broad range of tabular operations, including column mutation

Fluvio is a distributed event streaming platform and cloud-native streaming engine designed for collecting, persisting, and replicating real-time data streams across a distributed cluster. It functions as a real-time data pipeline for building stateful workflows that ingest, enrich, and export data between external sources and sinks. The platform is distinguished by its use of WebAssembly to execute compiled modules for in-line data transformations and filtering. This allows for the execution of custom business logic to reshape information in motion without requiring a restart of the cluster.

The OpenTelemetry Collector is a vendor-agnostic proxy and observability data pipeline that receives, processes, and exports traces, metrics, and logs. It functions as a telemetry ingestion gateway and multi-backend monitoring agent, translating various data formats into a standardized internal representation for consistent processing. The project distinguishes itself through a plugin-based component model, allowing the integration of custom receivers, processors, and exporters without modifying the core codebase. It utilizes a configurable pipeline system where telemetry flows through a sequ

Aidoku is a manga reader application and digital library manager. It serves as a modular content aggregator that allows users to discover, download, and read manga from various third-party sources and local files. The application utilizes a modular source plugin system to integrate external provider packages, enabling the ingestion of content from multiple third-party sources. It includes a sync engine that communicates with external tracking APIs to maintain consistent reading progress across different platforms. The system covers manga library management, including the ability to search fo

elasticsearch-dump is a command line tool for importing, exporting, and transferring data between Elasticsearch and OpenSearch instances. It functions as an index dump utility that saves documents, mappings, and analyzers to local files or standard output. The tool enables the movement of data between clusters using local files as an intermediary and can flatten nested JSON documents into CSV files for external analysis. It allows for the modification or anonymization of documents during the transfer process through the use of custom JavaScript functions. The utility covers data extraction a

Redis is a high-performance in-memory key-value store that functions as a distributed cache, message broker, and NoSQL database. It provides sub-millisecond read and write access to data stored in RAM and can operate as a vector database for indexing high-dimensional embeddings. The system supports a wide range of data storage and synchronization primitives, including the management of strings, hashes, lists, sets, and JSON documents. It enables real-time data operations through atomic transactions, hybrid persistence using snapshots and append-only logs, and high-availability configurations

DevLake is a DevOps data platform and analytics tool designed to orchestrate data pipelines that ingest, transform, and sync metadata from external development tools into a unified database. It functions as a system for collecting and normalizing data from source control, CI/CD pipelines, and issue trackers into a standardized schema to enable consistent software delivery analytics. The platform distinguishes itself by transforming tool-specific data into a common domain model, allowing for the calculation of engineering metrics via SQL. It provides specialized frameworks for measuring DORA m

Lux is a blockchain infrastructure project providing a distributed ledger and a smart contract platform. It utilizes a hybrid proof-of-work and proof-of-stake consensus mechanism to secure decentralized transactions and maintain network stability. The project features a peer-to-peer ecosystem that includes a decentralized file storage system for immutable data, a masternode network for governance, and a cross-chain asset bridge. It distinguishes itself with an ASIC-resistant hashing algorithm optimized for general-purpose CPUs and a decentralized order book for matching trades without a centr

Data-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets to prepare them for training large language and vision models. It functions as a distributed data pipeline engine that runs processing jobs across Ray clusters, handling billions of samples with automatic operator fusion and adaptive parallelism. The framework provides a library of operators that leverage large language models for semantic extraction, filtering, and data synthesis within processing pipelines. The project distinguishes itself through a YAML-based data recipe sys

Fast-F1 is a Python data library and telemetry analyzer designed for Formula 1. It serves as a programmatic API wrapper to retrieve and process timing data, session results, and high-frequency vehicle sensor metrics. The project provides specialized tools for motorsport performance benchmarking and visualization. It enables the comparison of vehicle telemetry, such as speed traces and gear shifts, and generates spatial track maps with speed and corner overlays. The library covers a broad range of data retrieval and analysis capabilities, including the extraction of racing schedules, champion

This project provides a collection of processed Chinese conversational datasets and preprocessing workflows designed for training and instruction tuning of large language models. It functions as a training corpus of cleaned, standardized Chinese text formatted as query-answer pairs. The repository includes a preprocessing pipeline and dataset aggregator that combine multiple public chat sources into unified files. These tools normalize text by converting traditional Chinese characters to simplified characters and transforming complex dialogue threads into a standardized sequence of single tur

Huginn is an open-source automation platform that functions as an event-driven task automator and webhook integration engine. It enables the creation of agents that monitor web data and automate tasks across various web services, operating as a self-hosted web scraper and JavaScript workflow orchestrator. The system uses a directed graph of event flows to route and transform data between external APIs. It differentiates itself by allowing custom JavaScript execution within workflows to modify data payloads and by integrating human-in-the-loop automation to insert manual judgment or data entry

immudb is a tamperproof database that maintains an immutable record of entries using cryptographic commit logging. It ensures verifiable database integrity by utilizing Merkle trees to generate membership and consistency proofs that detect unauthorized data alterations. The system employs a multi-model storage engine that unifies key-value, document, and relational data structures within a single immutable backend. It provides compatibility with the PostgreSQL wire protocol, allowing it to integrate with standard SQL clients, ORMs, and database tools. The project covers broad capabilities in

This project provides a suite of interfaces and tools for accessing electricity carbon intensity and production metrics. It includes an API for real-time and historical data, a geographic power data map for visualizing regional carbon intensity and renewable energy percentages, and a system for extracting datasets required for standardized greenhouse gas emissions reporting. The project features an interactive API sandbox that allows users to test requests and inspect data responses without writing code. It also includes mechanisms for institutional email verification to manage access to hist

XChange is a cryptocurrency exchange integration library and API wrapper that provides a unified interface for connecting to multiple cryptocurrency exchanges. It serves as a multi-exchange trading interface and a market data streamer, normalizing raw data from various providers into standardized data objects. The project distinguishes itself through an adapter-based normalization system and a reactive WebSocket streaming model designed to receive real-time price and volume updates while minimizing thread usage. It includes a dedicated authentication handler for managing API keys, request non

mmagic is a multimodal training pipeline and framework for generative AI, focusing on visual synthesis and restoration. It provides the infrastructure to build and train models for tasks such as text-to-image and text-to-video generation, 3D-aware content synthesis, and high-fidelity image translation using diffusion models and generative adversarial networks. The project distinguishes itself through specialized capabilities for generative model personalization, including techniques for fine-tuning subjects and styles. It also supports advanced visual manipulations such as latent space interp

Serial Studio is a desktop application for connecting to, decoding, visualizing, and recording data from hardware devices over multiple communication protocols. It functions as an embedded device debugging toolkit that ingests live data from Serial, Bluetooth, CAN, Modbus, MQTT, and network sockets into a unified dashboard, while also serving as a programmatic automation platform with over 320 commands exposed over TCP, gRPC, and MCP for external control. The application distinguishes itself through a scriptable frame pipeline that routes incoming bytes through configurable detection, decodin

This project is a comprehensive collection of country names and standardized ISO 3166-1 codes. It serves as a geographic reference repository that maps global nations to their respective international standards for use in data normalization and identification. The dataset is provided in multiple languages to support internationalization and is distributed in various file formats to ensure compatibility across different programming environments. The project covers capabilities for country metadata mapping, multilingual geographic labeling, and the retrieval of alpha-2 and alpha-3 codes for ad

TOML is a configuration file format designed for human readability and unambiguous mapping to hash tables. It serves as a standardized language for structured data, enabling consistent parsing and data exchange across diverse programming environments. The format distinguishes itself through a strict type-system specification that ensures data is interpreted identically regardless of the implementation. It utilizes a line-oriented lexical structure that supports both hierarchical organization through bracketed sections and compact inline embedding for nested objects. This approach allows for t

dbt-core is a command-line framework for transforming data within a warehouse using modular SQL and version control. It functions as a data transformation engine that enables users to define data structures and business logic through declarative configuration files, which the system then compiles into executable code. By managing complex data dependencies through a directed acyclic graph, it ensures that transformation tasks execute in the correct order while maintaining a manifest-driven state to track lineage and execution history. The project distinguishes itself through an adapter-based d

This project is a centralized API specification registry and governance framework. It serves as a source of truth for structured API schemas, gRPC definitions, and data models used to maintain consistency across a large ecosystem of services. The system functions as a multi-language client generator, transforming standardized interface definitions into executable source code and language-specific libraries. This process ensures that client libraries and technical documentation remain synchronized by deriving them directly from the original interface definitions. The project also covers the m

This repository is a collection of Jupyter notebooks providing reference implementations and templates for building, training, and deploying machine learning models using Amazon SageMaker. It serves as an example library for implementing model architectures and automating the machine learning lifecycle. The library provides practical patterns for machine learning training, data engineering, and model deployment. It includes implementation guides for MLOps, including workflows for model monitoring, lineage tracking, and hyperparameter tuning. The examples cover a broad range of capabilities i

Nextflow is a dataflow workflow engine and distributed computing framework used to build and execute data-intensive pipelines. It serves as a scientific workflow language that allows users to define reproducible data processing sequences, supporting any scripting language through shebang declarations. The system functions as a containerized pipeline orchestrator, utilizing container technologies to ensure software dependencies remain consistent across different environments. It decouples workflow logic from the underlying infrastructure, enabling the same pipeline to run on local machines, cl

Daft is a distributed dataframe library and multimodal data processor designed to handle large-scale structured and unstructured data. It functions as a vectorized execution engine that processes tables alongside images, audio, and video, utilizing a unified schema to manage diverse data types. The project distinguishes itself by combining distributed data engineering with large-scale AI inference. It provides an AI data pipeline for batch-optimizing model prompts and generating high-dimensional text embeddings, while utilizing zero-copy memory sharing to execute custom Python functions witho