4 Repos
Awesome GitHub RepositoriesLarge Data Streamers
Parsers for handling massive JSON files via streaming.
Distinguishing note: Focuses on memory-efficient streaming of large files rather than general parsing.
Explore 4 awesome GitHub repositories matching data & databases · Large Data Streamers. Refine with filters or upvote what's useful.
Awesome Large Data Streamers GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- jqlang/jq
jqlang/jq
34,901Auf GitHub ansehenThis project is a command-line processor designed for the parsing, filtering, and transformation of structured data streams. It functions as a declarative programming environment that treats data as immutable streams, allowing users to perform complex structural modifications through the composition of small, reusable functions. By utilizing a recursive tree traversal engine, the system enables the navigation, inspection, and modification of deeply nested hierarchical data structures. The engine distinguishes itself through a stream-oriented architecture that processes input records one by on
Parses massive JSON documents by streaming path-value pairs to minimize memory usage.
Cjq - request/request
request/request
25,542Auf GitHub ansehenThis is an HTTP client library used for sending and receiving network requests. It functions as an HTTP traffic replicator, a multipart form uploader, and an OAuth request signer, while also serving as an HTTP client capable of routing traffic through Unix domain sockets for local inter-process communication. The project distinguishes itself with the ability to import and parse HTTP Archive JSON files to reproduce recorded network traffic. It also provides cryptographic OAuth signing to secure API access using hashing algorithms and supports routing requests through Unix domain sockets using
Implements memory-efficient streaming of large network payloads to files or pipes.
JavaScript - piskvorky/gensim
piskvorky/gensim
16,361Auf GitHub ansehenGensim is a natural language processing toolkit designed for large-scale text analysis and the training of semantic vector embeddings. It provides a framework for identifying latent thematic structures within document collections and calculating semantic similarity between text segments using unsupervised statistical algorithms. The project is distinguished by its ability to handle datasets that exceed available system memory through incremental corpus streaming, which processes documents one at a time from disk. It utilizes sparse vector representations and dictionary-based token mapping to
Process documents one at a time from a collection to enable analysis of datasets that exceed available system memory.
Pythondata-miningdata-sciencedocument-similarity - jimmywarting/streamsaver.js
jimmywarting/StreamSaver.js
4,355Auf GitHub ansehenStreamSaver.js ist eine Bibliothek für beschreibbare Browser-Streams und ein clientseitiger File-Saver. Sie bietet einen Mechanismus, um Datenströme direkt in das lokale Dateisystem zu schreiben und dabei den standardmäßigen Blob-basierten Speicheransatz zu umgehen, um eine Erschöpfung des Browserspeichers zu vermeiden. Die Bibliothek konvertiert Browser-Streams in herunterladbare Dateien und ermöglicht so den Export großer Datensätze und Dateien, ohne das gesamte Payload im RAM puffern zu müssen. Sie unterstützt das direkte Schreiben von Echtzeit-Audio- und Videoaufnahmen auf die Festplatte, um Speicherüberläufe bei langen Aufnahmesitzungen zu verhindern.
Handles massive data streams by writing them directly to the filesystem to avoid exhausting memory.
JavaScriptfilesaverhtml5ram
