4 repository-uri
Awesome GitHub RepositoriesLarge Data Streamers
Parsers for handling massive JSON files via streaming.
Distinguishing note: Focuses on memory-efficient streaming of large files rather than general parsing.
Explore 4 awesome GitHub repositories matching data & databases · Large Data Streamers. Refine with filters or upvote what's useful.
Awesome Large Data Streamers GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- jqlang/jq
jqlang/jq
34,901Vezi pe GitHubThis project is a command-line processor designed for the parsing, filtering, and transformation of structured data streams. It functions as a declarative programming environment that treats data as immutable streams, allowing users to perform complex structural modifications through the composition of small, reusable functions. By utilizing a recursive tree traversal engine, the system enables the navigation, inspection, and modification of deeply nested hierarchical data structures. The engine distinguishes itself through a stream-oriented architecture that processes input records one by on
Parses massive JSON documents by streaming path-value pairs to minimize memory usage.
Cjq - request/request
request/request
25,542Vezi pe GitHubThis is an HTTP client library used for sending and receiving network requests. It functions as an HTTP traffic replicator, a multipart form uploader, and an OAuth request signer, while also serving as an HTTP client capable of routing traffic through Unix domain sockets for local inter-process communication. The project distinguishes itself with the ability to import and parse HTTP Archive JSON files to reproduce recorded network traffic. It also provides cryptographic OAuth signing to secure API access using hashing algorithms and supports routing requests through Unix domain sockets using
Implements memory-efficient streaming of large network payloads to files or pipes.
JavaScript - piskvorky/gensim
piskvorky/gensim
16,361Vezi pe GitHubGensim is a natural language processing toolkit designed for large-scale text analysis and the training of semantic vector embeddings. It provides a framework for identifying latent thematic structures within document collections and calculating semantic similarity between text segments using unsupervised statistical algorithms. The project is distinguished by its ability to handle datasets that exceed available system memory through incremental corpus streaming, which processes documents one at a time from disk. It utilizes sparse vector representations and dictionary-based token mapping to
Process documents one at a time from a collection to enable analysis of datasets that exceed available system memory.
Pythondata-miningdata-sciencedocument-similarity - jimmywarting/streamsaver.js
jimmywarting/StreamSaver.js
4,355Vezi pe GitHubStreamSaver.js este o bibliotecă de fluxuri inscriptibile (writable streams) pentru browser și un instrument de salvare a fișierelor client-side. Oferă un mecanism pentru a scrie fluxuri de date direct în sistemul de fișiere local, ocolind abordarea standard de salvare bazată pe blob-uri pentru a evita epuizarea memoriei browserului. Biblioteca convertește fluxurile din browser în fișiere descărcabile, permițând exportul seturilor de date de mare volum și al fișierelor mari fără a stoca întregul payload în RAM. Suportă scrierea capturilor audio și video în timp real direct pe disc pentru a preveni depășirea memoriei în timpul sesiunilor lungi de înregistrare.
Handles massive data streams by writing them directly to the filesystem to avoid exhausting memory.
JavaScriptfilesaverhtml5ram
