Awesome GitHub RepositoriesLarge Dataset Streaming

Vegeta is an HTTP load testing tool and library designed to measure the performance and stability of web services. It functions as a command-line utility, a programmable package for integration into other applications, and a distributed load generator capable of splitting request rates across multiple machines. The tool is distinguished by its constant-rate request scheduler, which dispatches requests at a fixed frequency regardless of target response times. It employs lazy target streaming to maintain low memory usage during large tests and uses a binary-encoded storage format to minimize di

Excelize is a Go library designed for reading, writing, and modifying Microsoft Excel files in XML-based formats. It functions as a spreadsheet file parser and generator that enables the programmatic extraction and modification of data. The library includes a streaming spreadsheet processor to handle massive datasets incrementally, preventing system memory exhaustion during large-scale read and write operations. It also provides a chart generator to convert worksheet values or external data sources into visual representations within the spreadsheet. Beyond core file processing, the project c

Cesium is a JavaScript geospatial visualization library and 3D globe engine designed to render world-scale environments and precision spatial data. It functions as a web-based mapping tool that displays hardware-accelerated 3D globes and 2D maps directly in a browser without requiring external plugins. The project operates as a 3D tiles renderer, utilizing the 3D Tiles open standard to stream and display large-scale geospatial datasets. It enables the visualization and analysis of high-accuracy spatial information across global environments. The library covers a broad range of capabilities,

PapaParse is a delimited text processing library that converts CSV files into JSON objects or arrays. It provides a suite of tools for parsing delimited text and transforming structured data objects back into CSV formats through bidirectional serialization. The library is characterized by its ability to process massive datasets using incremental streaming and chunk-based processing to prevent memory overload. It includes an automatic delimiter detector to identify separator characters without manual configuration and utilizes web workers to offload parsing logic to background threads, keeping

This project is a Node.js client for PostgreSQL databases, providing a protocol parser to translate raw binary streams into JavaScript objects. It serves as a driver for executing queries, managing data, and integrating Node.js applications with PostgreSQL backends. The library includes a connection pool manager to reduce network overhead by caching reusable connections and a result streamer that uses cursors to retrieve large datasets incrementally. It also functions as an event listener for subscribing to asynchronous server-side notifications to trigger real-time application events. Broad

This project provides a lossless compression algorithm and a byte-oriented compression library designed for high-speed data reduction and maximum decompression speed. It functions as a stream-oriented compression engine, a software library for encoding and decoding data blocks, and a command-line tool for managing interoperable compressed frames. The system distinguishes itself through the use of predefined pattern dictionaries to improve compression ratios for small data sets and small packets. It supports multiple processing modes, including high-speed block compression for minimal latency

This project is a framework for the efficient serialization and deserialization of data structures. It provides a unified, macro-based interface that automates the conversion of complex internal objects into standardized formats and reconstructs them from raw input streams or buffers. By leveraging compile-time code generation, the library minimizes manual implementation overhead while ensuring consistent logic across diverse data types. The framework distinguishes itself through a format-agnostic data model and a visitor-based parsing architecture that decouples data structures from specific

This project is a collection of deep learning tutorials and practical implementations using TensorFlow. It provides a neural network implementation guide through code examples designed for research-oriented deep learning. The repository covers supervised and unsupervised learning workflows, including the development of sequence models for language processing and chatbots. It includes specific examples for image style transfer and the use of autoencoders for feature extraction. The project also provides demonstrations for managing large-scale datasets using binary record formats and streaming

3FS is a distributed file system and RDMA storage cluster designed for high-performance AI training and inference workloads. It functions as a strongly consistent storage layer that utilizes a disaggregated architecture to pool SSDs and memory resources across multiple nodes. The system provides specialized storage implementations including an AI training checkpoint store for parallel state preservation and a distributed key-value cache store for decoder layer vectors to optimize inference processing. It ensures data integrity through chain replication and apportioned query distribution. The

Miller is a command-line data processor used for filtering, transforming, and aggregating name-indexed tabular data. It functions as a tool for querying and reshaping records across multiple file formats, serving as a converter between CSV, JSON, and YAML. The tool distinguishes itself by using a name-indexed data model, allowing users to manipulate fields by name rather than numeric position. It utilizes single-pass streaming algorithms to compute statistics and summaries on large datasets that exceed available system memory. Its capabilities cover data transformation and analysis, includin

Jackson is a Java data binding framework and multi-format data serializer used to translate data structures into native language objects. It functions as a JSON data binding library and a streaming parser that reads and writes data as discrete tokens to process large datasets with minimal memory. The project distinguishes itself through a bytecode serialization accelerator that replaces standard reflection with generated bytecode to increase data binding speed. It employs a module-based extensibility model to support a wide range of formats beyond JSON, including XML, YAML, CSV, TOML, and bin

Hub is a multimodal AI data lake and vector database designed for storing and querying embeddings, text, audio, and images. It functions as a dataset version control system and a machine learning data streaming engine to support large-scale model training. The system utilizes a serverless PostgreSQL vector store to index high-dimensional embeddings for semantic search. It provides a visual interface for inspecting multimodal datasets and viewing annotations such as bounding boxes and masks. The platform handles cloud-agnostic storage synchronization and implements lazy, compressed data strea

DeepLake is AI data infrastructure consisting of a multimodal data lake, a hybrid search engine, and a serverless vector database. It provides a PostgreSQL-based AI data runtime that combines multimodal storage with streaming pipelines to load and shuffle datasets from cloud storage directly into deep learning training pipelines. The system utilizes lazy indexing to store and slice images, audio, and video without loading entire files into memory. It enables retrieval-augmented generation by persisting high-dimensional embeddings in a serverless vector store and implementing hybrid search tha

LanceDB is a vector database and columnar data store designed to function as a versioned dataset manager and vector search engine. It serves as a high-performance backend for indexing and retrieving high-dimensional embeddings, providing the foundation for machine learning data pipelines. The system distinguishes itself through a combination of cloud-native object storage and immutable version tracking, allowing for data time-travel and reproducible AI experiments. It integrates hybrid search capabilities, merging dense vector similarity with BM25 full-text search and SQL-like scalar filters

This project is a PostgreSQL client library and SQL query builder for JavaScript and TypeScript. It provides a low-level database driver and connection manager to handle database sessions, along with a logical replication client for monitoring real-time changes. The library distinguishes itself with a high-performance bulk data streamer that utilizes the database copy command for importing and exporting large datasets. It also implements a logical replication protocol to facilitate real-time database synchronization through change subscriptions and channel-based notifications. The toolset co

jc is a tool that transforms plain-text results from command-line utilities, system tools, log formats, and text tables into structured JSON data. It functions as a structured data transformer capable of converting various file formats, including CSV, INI, XML, and YAML, into JSON representations for programmatic use. The project includes a collection of specific parsers for Unix commands and system tools such as df, blkid, and various package managers. It also features specialized converters for web server logs, Common Log Format, and Common Event Format strings. The tool covers broad capab

Brain is a JavaScript library for building, training, and running feed-forward neural networks. It implements a multilayer perceptron model designed for pattern recognition and function approximation. The library includes a standalone inference engine that converts trained models into portable JavaScript functions. This allows predictions to be executed in browser or Node.js environments without requiring the original library dependencies. The system supports persistent model management through JSON serialization for saving and loading network weights. It also provides a streaming mechanism

language-ext is a functional programming framework for C# that provides a suite of immutable data structures and monadic types. It enables the implementation of pure functional programming patterns, utilizing containers to manage side effects, optional values, and error handling. The library is distinguished by its advanced concurrency and state management tools, including a software transactional memory system and lock-free atomic references. It also provides specialized utilities for distributed systems, such as vector clocks for causality tracking and deterministic data conflict resolution

aws-sam-local ist ein Docker-basierter Emulator, der zum Ausführen und Debuggen von AWS Lambda-Funktionen und API-Gateway-Konfigurationen auf einem lokalen Rechner verwendet wird. Er bietet eine lokale Laufzeitumgebung, die die AWS Lambda-Runtime nachahmt und die Verifizierung von Funktionscode sowie serverlosen Anwendungs-Mocks vor der Cloud-Bereitstellung ermöglicht. Das Projekt nutzt einen Runtime-Orchestrator zur Verwaltung ephemerer Container, die die isolierte Ausführungsumgebung von Cloud-Funktionen replizieren. Es enthält einen Serverless-API-Gateway-Simulator und Event-Mocking-Tools, um Payloads aus Quellen wie Kafka und anderen Cloud-Triggern zu synthetisieren. Das Tool unterstützt den breiteren Entwicklungszyklus für das Serverless Application Model, indem es lokales Prototyping von serverlosen APIs und die Simulation verschiedener Cloud-Event-Quellen ermöglicht. Es bietet eine Debugging-Umgebung zur Identifizierung von Fehlern in serverlosen Funktionen mit voller Kontrolle über den Ausführungsstatus.

This project is a Rust-based AI agent framework and tool orchestrator that provides a command-line interface for interacting with large language models. It functions as an AI tool orchestrator that routes client requests to language servers and manages the planning and handoffs between specialized agents to solve complex tasks. The system distinguishes itself as a language porting validator, using deterministic mocks and specifications to verify feature parity between different language implementations of a codebase. It further extends agent capabilities by acting as a Model Context Protocol