30 مستودعات
Streaming parsers specialized for incremental processing of large JSON datasets and concatenated streams.
Distinct from Streaming Parsers: Distinct from general Streaming Parsers: focuses on JSON-specific incremental processing rather than generic event-driven data streams.
Explore 30 awesome GitHub repositories matching data & databases · JSON. Refine with filters or upvote what's useful.
The Google Workspace CLI is a command-line interface and Google API client designed to automate tasks across Google Workspace services. It functions as a cloud productivity automator that uses the Google Discovery Service to dynamically generate command structures and parameter requirements at runtime. The project distinguishes itself by providing a specialized AI agent toolset, exposing a server over standard input and output to provide structured tool definitions and skills for AI clients. It includes security layers for AI content sanitization to protect against prompt injection and utiliz
Outputs paginated API datasets as a continuous stream of newline-delimited JSON objects for piping.
Fastjson is a Java data binding framework and serialization library designed to convert objects to JSON strings and parse JSON data into typed objects. It functions as a JSON parser and stream processor capable of transforming JSON strings into data structures. The project emphasizes high performance JSON processing and memory management, specifically through the use of a pipeline to stream oversized JSON objects and texts to prevent memory exhaustion. It provides capabilities for JSON data serialization and deserialization workflows, including custom JSON data mapping and the ability to def
Implements streaming parsers to incrementally process oversized JSON datasets and prevent memory exhaustion.
simdjson is a high-performance, header-only C++ library designed for parsing, querying, and serializing JSON data with minimal memory overhead. It functions as a hardware-aware data processing engine that leverages vector instructions to achieve gigabyte-per-second parsing speeds. By detecting host processor capabilities at runtime, the library automatically selects the most efficient instruction sets to accelerate structural analysis and validation. The library distinguishes itself through a focus on extreme efficiency and resource management. It utilizes memory mapping and padded buffer ali
A tool for processing concatenated JSON documents and large data streams incrementally without loading entire files into memory.
This project is a REST-to-gRPC API gateway and JSON reverse proxy that translates RESTful HTTP requests into gRPC service calls. It functions as a protocol buffer proxy generator, providing the tools necessary to bridge JSON-based HTTP traffic with backend gRPC servers. The system distinguishes itself by automating the creation of reverse-proxy servers and stubs through protobuf-driven code generation. It also includes a dedicated OpenAPI specification generator that produces OpenAPI v2 and v3 documents from gRPC service definitions and annotations. The project covers a broad range of integr
Maps gRPC server streams to newline-delimited JSON streams for compatible HTTP delivery.
WeKnora is a multi-tenant retrieval-augmented generation (RAG) knowledge platform and autonomous AI agent framework. It transforms raw documents into queryable knowledge bases and integrates large language models with vector databases to provide grounded AI responses. The system also functions as a Model Context Protocol (MCP) tool server, exposing knowledge search and agentic capabilities to external AI clients. The platform distinguishes itself through an autonomous agent framework that utilizes iterative reasoning, tool calling, and web search to solve multi-step tasks. It implements a sta
Uses line-delimited JSON streams to deliver AI responses and process updates for incremental processing.
gjson is a Go JSON parser designed for schema-less reading and value extraction. It allows for the retrieval of specific data from JSON documents using dot-notation paths without requiring the definition of predefined Go structs. The library provides tools for path-based querying, including the use of wildcards and index-based queries to locate data within objects and arrays. It also functions as a JSON lines processor, treating multi-line documents as arrays to iterate and query individual entries. Additional capabilities include converting JSON values into native Go types such as strings,
Treats multi-line documents as arrays to iterate and query individual JSON entries.
RapidJSON is a high-performance C++ library used for parsing and generating JSON data. It provides both document object model and stream-based interfaces to transform JSON strings into structured data and vice versa. The library includes a JSON schema validator to verify that documents conform to predefined rules and a Unicode transcoder for converting strings between UTF-8, UTF-16, and UTF-32 encodings. It also supports relaxed parsing for non-standard JSON containing comments or trailing commas. Additional capabilities cover JSON pointer navigation for locating specific values and string s
Provides a stream-based SAX parser that processes JSON as a series of events to minimize memory overhead.
RapidJSON is a header-only C++ library designed for high-performance parsing, generation, and manipulation of JSON data. It functions as a dual-mode engine, providing both an in-memory document object model for tree-based manipulation and a stream-based interface for event-driven processing. The library is built to minimize memory footprint and maximize execution speed, making it suitable for resource-constrained environments. The library distinguishes itself through advanced memory management and optimization techniques, including in-situ parsing that modifies input buffers directly to elimi
Handles large data sources by processing input and output through buffers.
Got is a promise-based HTTP request library for Node.js that supports HTTP/2 and streaming. It provides a system for making network requests with a focus on asynchronous control flow and type-safe API client development. The library is distinguished by its middleware-based request lifecycle, which uses interceptors and plugins to modify request options and response data. It includes a configurable automatic retry mechanism with backoff strategies, a built-in HTTP response cache, and a cookie-jar system for maintaining persistent sessions. Broad capabilities cover data handling through duplex
Processes request and response bodies as duplex streams to handle large datasets without memory exhaustion.
Gron is a command line utility that transforms nested JSON data into a flat list of path-value assignments. This process converts hierarchical structures into line-based statements, mapping every leaf value to its absolute path to make the data compatible with standard text-processing tools. The tool allows for the bidirectional transformation of data, enabling the reconstruction of original nested JSON objects from flattened path assignments. It can ingest JSON from local files, standard input, or remote URLs, with the ability to route network traffic through proxy servers. The utility supp
Employs memory-efficient streaming to process large JSON inputs without loading entire files into memory.
Instructor is a schema enforcement and validation library designed to transform language model outputs into structured, type-safe data formats. It functions as a validation layer that uses Pydantic to ensure model responses conform to specific data models, acting as a tool for forcing large language models to return data in predefined schemas. The project differentiates itself through a recursive error-feedback loop that automatically retries requests when structural errors occur, passing validation failure messages back to the model to guide corrections. It also includes a streaming parser c
Provides a parser that processes partial JSON fragments in real time as they are generated by a language model.
Moshi is a JSON serialization library and parser for Kotlin and Java. It functions as a reflectionless JSON encoder that converts typed objects to JSON strings and parses JSON data back into language objects. The library distinguishes itself through compile-time adapter generation, which removes the performance overhead associated with runtime reflection. It also provides a polymorphic JSON mapper that uses type identifiers to resolve and instantiate specific subclasses of a common base type. The framework supports custom adapter definitions for specialized type conversion, including nullabi
Provides the ability to peek at tokens in a JSON stream without consuming them.
This project is a structured Node.js programming course and educational guide designed to teach JavaScript backend development. It provides a sequence of workshops and interactive tutorials that focus on the fundamentals of the Node.js runtime and its core modules. The material emphasizes asynchronous programming, specifically covering non-blocking I/O, callback patterns, and event-driven architecture. It includes a practical exploration of the core API for managing network applications, file system operations, and binary data. The curriculum covers module management and dependency resolutio
Provides instruction on using memory buffers to efficiently process binary data and large datasets.
Jackson is a Java data binding framework and multi-format data serializer used to translate data structures into native language objects. It functions as a JSON data binding library and a streaming parser that reads and writes data as discrete tokens to process large datasets with minimal memory. The project distinguishes itself through a bytecode serialization accelerator that replaces standard reflection with generated bytecode to increase data binding speed. It employs a module-based extensibility model to support a wide range of formats beyond JSON, including XML, YAML, CSV, TOML, and bin
Implements a low-level streaming parser for incremental processing of large JSON datasets.
Sonic is a high-performance Go JSON serialization library that provides tools for encoding and decoding native data structures. It functions as a JIT-accelerated encoder, a JSON AST parser, a stream processor, and a lazy decoder. The project utilizes just-in-time machine code generation to optimize the encoding of large data schemas and employs a JIT assembler to maximize serialization and deserialization speeds. It features a precompiled schema warmup process to prevent latency spikes during initial execution and leverages SIMD hardware instructions for accelerated parsing. The library cove
Writes native objects as JSON to output streams with configurable indentation and HTML escaping.
Proxygen is a collection of C++ libraries for building high-performance HTTP servers and clients. It provides a protocol parser that converts raw network bytes into high-level transaction objects and includes a network stack for processing web traffic over the QUIC transport protocol. The project implements a layered protocol abstraction and a QUIC-based transport integration to support multiple versions of the HTTP standard, including HTTP/3. It utilizes state-machine based parsing and an event-driven I/O loop to manage concurrent network connections. The library covers asynchronous buffer
Utilizes non-blocking memory buffers to efficiently queue and stream data between the network socket and the application.
WeixinBot is a framework for WeChat account automation and bot development. It provides a programmable interface to monitor incoming messages and trigger automated actions within the WeChat ecosystem. The system emulates a browser session using a WebSocket-based protocol and establishes identity through QR code authentication. It maintains session-state persistence to keep accounts active without repeated verification. The project covers message processing for text, images, voice notes, and location data, and supports the transmission of emojis and links. It includes utilities for user conta
Provides background downloading of binary image and voice data to maintain system responsiveness.
هذا المشروع عبارة عن مجموعة بيانات واسعة النطاق للرسومات اليدوية، توفر ملايين الرسومات المتجهة (vector) والصور النقطية (bitmaps) المؤرخة لتدريب نماذج تعلم الآلة. تعمل كمرجع لتدريب الرؤية الحاسوبية ومجموعة بيانات للشبكات العصبية، وتتكون من رسومات بشرية مصنفة تُستخدم لتطوير خوارزميات تصنيف والتعرف على الصور. تتوفر مجموعة البيانات كمرجع للرسومات المتجهة يتميز بتسلسلات ضربات القلم والبيانات الوصفية، بالإضافة إلى مصفوفات numpy المعالجة. تدعم هذه الموارد تطوير مصنفات الرسم ودراسة أنماط الرسم البشري. يتم توفير البيانات بتنسيقات متعددة، بما في ذلك بيانات المتجهات الخام بتنسيق JSON مفصول بأسطر جديدة، وتسلسلات متجهة طبيعية، وصور نقطية بتدرج الرمادي. تتضمن قدرات للتقسيم القائم على الفئات وتوسيع الإحداثيات لضمان الاتساق عبر عينات مختلفة.
Stores large datasets as separate JSON objects per line to allow efficient streaming and partial file reading.
jnv is an interactive terminal application for querying and filtering JSON data using jq expressions. It combines a keyboard-driven JSON browser with a real-time jq filter editor, allowing users to navigate, expand, and collapse JSON structures while simultaneously writing and previewing filter results. The tool reads JSON from files or standard input, including JSON Lines format, and provides immediate visual feedback as filters are typed. The application distinguishes itself by integrating jq filter development with live preview and auto-completion, suggesting completions for identifiers, o
Reads JSON input incrementally from files or stdin, handling multiple JSON objects per stream including JSON Lines.
xcodebuildmcp is a Model Context Protocol server that exposes Xcode build, test, and device management tools for AI coding agents to automate iOS and macOS development workflows. It operates as a background daemon per workspace, communicating tool requests and responses over standard input/output using JSON-RPC messages, and streams progress and results as newline-delimited JSON objects for machine parsing. The project provides an interactive setup wizard and file-based client configuration to install skill files into predefined directories for supported AI coding clients. It manages the full
Emits progress and results as newline-delimited JSON objects for machine parsing and shell piping.