Awesome GitHub RepositoriesLarge Data Streamers

This project is a command-line processor designed for the parsing, filtering, and transformation of structured data streams. It functions as a declarative programming environment that treats data as immutable streams, allowing users to perform complex structural modifications through the composition of small, reusable functions. By utilizing a recursive tree traversal engine, the system enables the navigation, inspection, and modification of deeply nested hierarchical data structures. The engine distinguishes itself through a stream-oriented architecture that processes input records one by on

This is an HTTP client library used for sending and receiving network requests. It functions as an HTTP traffic replicator, a multipart form uploader, and an OAuth request signer, while also serving as an HTTP client capable of routing traffic through Unix domain sockets for local inter-process communication. The project distinguishes itself with the ability to import and parse HTTP Archive JSON files to reproduce recorded network traffic. It also provides cryptographic OAuth signing to secure API access using hashing algorithms and supports routing requests through Unix domain sockets using

Gensim is a natural language processing toolkit designed for large-scale text analysis and the training of semantic vector embeddings. It provides a framework for identifying latent thematic structures within document collections and calculating semantic similarity between text segments using unsupervised statistical algorithms. The project is distinguished by its ability to handle datasets that exceed available system memory through incremental corpus streaming, which processes documents one at a time from disk. It utilizes sparse vector representations and dictionary-based token mapping to

StreamSaver.js هي مكتبة دفق قابلة للكتابة للمتصفح وأداة لحفظ الملفات من جانب العميل. توفر آلية لكتابة تدفقات البيانات مباشرة إلى نظام الملفات المحلي، متجاوزة نهج الحفظ القياسي القائم على الكائنات (blob) لتجنب استنفاد ذاكرة المتصفح. تقوم المكتبة بتحويل تدفقات المتصفح إلى ملفات قابلة للتنزيل، مما يتيح تصدير مجموعات البيانات الضخمة والملفات الكبيرة دون تخزين الحمولة بالكامل في ذاكرة الوصول العشوائي (RAM). كما تدعم كتابة تسجيلات الصوت والفيديو في الوقت الفعلي مباشرة إلى القرص لمنع تجاوز سعة الذاكرة أثناء جلسات التسجيل الطويلة.