9 个仓库
Utilities for fetching web content via HTTP requests to be used as sources in data pipelines.
Distinct from Data Retrieval Pipelines: Focuses on HTTP-specific retrieval as a pipeline source, whereas Data Retrieval Pipelines is the broader system for normalization.
Explore 9 awesome GitHub repositories matching data & databases · HTTP Content Retrievers. Refine with filters or upvote what's useful.
Falcor is a JavaScript library that models remote data as a single virtual JSON graph, providing a path-based query engine for efficient client-side data retrieval and updates. It represents multiple remote data sources as a unified document where entities are accessed via globally unique identity paths. The system distinguishes itself by treating the remote data model as a virtual JSON resource, allowing the client to query specific paths without managing individual endpoints. It uses a reference-aware graph model to handle many-to-many relationships and prevents data duplication. Network ef
Fetches JSON graph data from remote servers using HTTP or HTTPS protocols.
LearnPython is a programming tutorial consisting of a collection of practical code examples used to demonstrate Python language features and programming patterns. It serves as a comprehensive learning resource that implements core language concepts through functional code. The project provides specialized guides and samples covering several key domains. These include asynchronous network programming with event loops and coroutines, data visualization using numerical datasets for 2D and 3D plots, and web scraping for fetching content and automating login flows. It also features instructions on
Fetches web content via HTTP requests and processes responses as plain text or raw bytes.
ArduinoJson is a C++ library for parsing and manipulating JSON data and MessagePack binary streams on microcontrollers with limited memory and processing power. It provides the core primitives necessary for embedded data serialization and parsing, enabling devices to exchange structured data over serial or network interfaces. The library is distinguished by its focus on microcontroller memory management, employing strategies such as pool-based allocation, string deduplication, and non-owning string views to minimize RAM usage. It further optimizes for constrained environments by allowing cons
Provides utilities to fetch JSON documents from remote servers via HTTP requests.
This project is a Go shell scripting library and framework designed for writing automation scripts and CLI tools. It provides a concurrent data pipeline system for chaining sources, filters, and sinks to process text and JSON streams. The library distinguishes itself through a comprehensive toolkit for shell-like operations, including a text processing engine for regular expression filtering and frequency analysis, a filesystem utility toolkit for recursive search and path manipulation, and an integrated HTTP client wrapper for building data pipelines that fetch web content. The capability s
Provides tools for retrieving web data using GET or POST requests to feed into processing pipelines.
CppGuide is a curated collection of educational resources and practical guides focused on C++ server development, Linux kernel internals, concurrent programming, network protocols, and security exploitation. It provides structured learning paths for backend developers, covering everything from interview preparation to building high-performance network servers and understanding operating system fundamentals. The guide distinguishes itself by offering in-depth, hands-on tutorials that walk through real-world implementations, including building a Redis-like server from scratch, designing custom
Retrieves web content via HTTP requests for testing and debugging endpoints.
该项目是一个 Python 网络爬虫库和自动化数据收集套件。它提供了从网站提取结构化数据的工具,实现了用于导航站点链接的网络爬虫,并能解析 HTML DOM 结构以隔离特定的元素和属性。 该工具包包含一个用于处理非结构化文本和清洗原始 Web 内容以提取有意义信息的流水线。它还具备图像数据提取功能,并支持集成外部 API 以从远程端点检索结构化数据。 该系统涵盖了广泛的功能领域,包括自动化 Web 数据提取、网络爬虫工作流,以及使用代理和验证码(CAPTCHA)求解器绕过爬取障碍的技术。
Provides HTTP content retrievers to fetch raw web content as sources for data pipelines.
This project is a Go library and command-line utility designed for the retrieval and local archival of remote video content. It provides a programmatic interface for fetching media streams, allowing users to extract metadata and download video files directly to local storage. The library distinguishes itself through its ability to resolve playback restrictions by performing algorithmic transformations on obfuscated authentication tokens. This signature decryption process enables the tool to bypass standard access limitations, while its interface-driven design allows for the selection of speci
Provides utilities for fetching web content via HTTP requests to be used as sources in data pipelines.
Tabletop is a JavaScript library and data parser designed to retrieve data from public Google Sheets and convert it into structured JSON objects. It functions as a client-side tool for fetching remote spreadsheet data and transforming rows into lists of objects or arrays for use in web applications. The library enables the use of Google Sheets as a lightweight database, allowing for dynamic content management where application data can be updated by editing a spreadsheet. It supports selective worksheet retrieval to limit the amount of transferred data and provides programmatic interfaces for
Fetches raw spreadsheet content from remote URLs via asynchronous HTTP requests to use as data sources.
GPAC is an open-source multimedia framework built around a pluggable filter graph pipeline, where modular processing units called filters connect into a directed graph to handle media workflows. At its core, the framework centers all media packaging and manipulation on the ISO Base Media File Format (ISOBMFF), with specialized tools for reading, writing, fragmenting, and encrypting MP4 and related containers. It also provides a declarative scene graph composition system for describing interactive multimedia scenes using MPEG-4 BIFS, X3D, SVG, or VRML syntax, alongside a hardware-accelerated re
Downloads remote media content via HTTP and feeds it as raw data blocks into the processing pipeline.