Awesome GitHub RepositoriesIterator-Based Pagination

Octokit.js is a GitHub API SDK used to interact with GitHub via REST and GraphQL interfaces. It serves as a client for executing typed queries and mutations, managing authenticated REST requests, and handling signed webhooks across Node.js, Deno, and browser environments. The library features a plugin-based request pipeline and an adapter-based HTTP client, allowing the interception of requests and responses across different runtimes. It employs strategy-based authentication to separate credential management from request logic, supporting static tokens, OAuth flows, and GitHub App installatio

PyGithub is a Python library that serves as an object-oriented wrapper for the GitHub REST API. It functions as a specialized client that translates API endpoints into a hierarchy of Python classes, allowing for the programmatic management of GitHub repositories, user profiles, and organizations. The library manages the serialization of requests and responses, converting JSON data into structured object attributes. It utilizes lazy-loading for property retrieval and employs generators to handle pagination when iterating over large remote datasets. This toolkit enables the automation of GitHu

JMComic-Crawler-Python is a high-performance asynchronous web scraper and API client designed to programmatically retrieve images and metadata from a comic hosting service. It functions as a media archiving tool for batch downloading albums and chapters, automating the process of saving content to a local filesystem. The project is distinguished by its ability to reverse server-side pixel obfuscation, using a decryption tool to reconstruct sliced and shuffled images. To maintain stable connectivity, it utilizes a network bypass utility featuring dynamic domain rotation and proxy routing to ci

The AWS SDK for PHP is a software development kit that provides HTTP client classes for every supported Amazon Web Service, enabling PHP applications to send authenticated requests and receive structured, typed response objects. It includes a credential resolution chain that automatically locates credentials from environment variables, instance profiles, or configuration files, and supports promise-based asynchronous execution for running multiple API calls concurrently to improve throughput. The SDK distinguishes itself through a middleware pipeline architecture that allows interception and

X-Ray 是一个 Web 抓取框架和异步 Web 爬虫，旨在从网站中提取结构化数据。它作为一个 HTML 数据提取器，使用 CSS 样式选择器将原始页面内容转换为定义的模式。 该项目实现了一个能够执行 JavaScript 以渲染动态内容的无头浏览器爬虫。它通过广度优先爬取策略和自动分页发现来处理网站内容发现，以遍历多页结果集。 该框架使用并发限制的请求队列和请求速率控制来管理 Web 数据管线，以调节传出的网络调用。提取的结果通过基于流的数据持久化进行处理，以在不占用系统内存的情况下处理大数据集。

X-ray 是一个无头浏览器 Web 爬虫和 HTML 内容抓取工具，旨在从网站中提取结构化数据。它作为一个基于流的数据抓取器和结构化数据提取器，使用选择器将 HTML 中的文本和属性检索为嵌套对象或数组。 该项目包含一个请求速率控制器，通过并发限制、节流和超时来管理网络流量。它通过无头浏览器渲染 JavaScript 来处理动态网站抓取，并使用广度优先链接跟踪和分页管理执行自动化网站爬取。 该系统提供了一个数据管线，将函数式值转换应用于原始字符串，并将结果写入可读流，以防止在大规模 Web 抓取作业期间出现内存溢出。

This project is a JavaScript client library and SDK for interacting with the Notion REST API. It provides typed methods and utilities to programmatically manage pages, databases, and workspace resources within a Notion environment. The SDK includes tools for implementing OAuth 2.0 authorization flows to grant third-party application access to workspace content. It features mechanisms for handling cursor-based pagination via asynchronous iterators and implements exponential back-off retries to manage rate limits and transient server errors. The library covers a broad range of capabilities, in

dlt 是一个 Python 数据摄取工具和 ETL 流水线框架，旨在从不同来源获取数据并将其持久化到结构化目标中。它作为一个模式推断引擎，可自动检测数据类型并将嵌套的 JSON 结构扁平化为关系表，将数据从源端移动到数据湖、数据仓库或向量数据库。 该项目通过 AI 驱动的流水线生成脱颖而出，利用大语言模型为 REST API 构建提取代码和连接器。它还支持多模态向量存储和向量数据库的专门填充，以支持 AI 和机器学习应用。 该框架涵盖了广泛的功能，包括自动化模式演进、通过状态跟踪进行增量数据加载，以及通过强制执行数据契约进行数据质量验证。它提供了用于关系数据规范化、加载前后转换的工具，以及针对 SQL 数据库和云对象存储的多种目标适配器。 可观测性通过流水线执行仪表板、列血缘跟踪以及使用基于内容的哈希进行模式版本验证来处理。

snscrape 是一款基于 Python 的社交媒体 Web 爬虫和抓取工具，旨在在不使用官方 API 的情况下从社交网络中提取公开帖子、个人资料和标签。它作为一个归档工具和开源情报数据收集实用程序，允许收集公开可用信息以调查趋势和人物。 该工具促进了用于研究和归档目的的社交媒体数据提取，实现了对话和用户活动历史记录的创建。它支持学术社交分析的工作流，并将大量元数据和消息导出到本地文件。 功能包括抓取各种社交网络平台并限制提取结果数量的能力。该系统可以将发现的项目导出为 URL 列表或包含内容和时间戳的详细文件。

This is a typed server-side library and payment gateway SDK for integrating Stripe into Node.js applications. It provides a typed client to manage payments, customers, and subscriptions, while offering specialized tools for executing secure financial transactions and managing billing resources. The library distinguishes itself through an idempotent API client that prevents duplicate operations using idempotency keys and exponential backoff retry logic. It includes a webhook signature validator to verify that incoming HTTPS event notifications are authentic and an async-iterator pagination wra

AWS Java SDK 是一个用于使用 Java 语言管理和交互 AWS 云基础设施及服务的编程接口和客户端库。它作为一个 API 封装器，通过一组标准化的 Java 类将应用连接到云存储、数据库和计算资源。 该库提供非阻塞 I/O 模型和基于 Future 的请求处理，以支持高吞吐量的异步云操作。它包含用于无服务器性能优化的机制，以减少这些环境中的初始启动时间和冷启动。 该 SDK 管理云服务交互的完整生命周期，包括针对多部分结果集的自动响应分页，以及一个加载并刷新安全令牌的基于链的凭据解析系统。它还具有可插拔的 HTTP 传输层，允许在运行时用自定义实现替换默认的网络客户端。

该项目是一个 Node.js Web 抓取框架，旨在通过请求、解析和文档交互的程序化工作流自动化数据提取。它作为一个无头 Web 爬虫、HTTP 请求管理器以及 DOM 解析器和提取器。 该框架通过结合用于与动态内容交互的 JavaScript 执行引擎和利用 CSS 及 XPath 选择器的混合选择系统脱颖而出。它包括用于代理轮换和 Cookie 罐会话管理的专用中间件，以维护身份验证状态并管理自动化流量。 其更广泛的功能涵盖递归链接爬取、分页处理和 Web 表单自动化。该工具还提供流量管理功能，例如通过定时延迟进行请求速率限制和自定义 HTTP 标头配置。

WeiboSpider 是一个社交媒体爬虫，旨在从新浪微博平台提取用户资料、帖子和交互数据。它作为一个基于 Web 的数据爬虫，通过外部接口检索信息，而不是解析可视化前端。 该工具包括一个内容血缘追踪器，用于将分享的帖子追溯到其原始来源。它还具有一个社交参与度分析器，用于收集浏览量和嵌套评论线程，以衡量用户交互指标。 该系统提供了基于关键字的社交监控和搜索结果过滤功能，以跟踪特定主题随时间的变化。它通过基于分页的迭代和参与线程的递归遍历来管理大数据集。

Octokit.rb is a Ruby client library and REST API wrapper used to programmatically interact with GitHub. It provides a structured interface for automating workflows, managing repositories, and transforming raw HTTP responses into Ruby objects. The library differentiates itself through hypermedia-driven navigation, which uses URI templates and link relations to discover resources dynamically. It also implements automated result pagination to combine paginated responses into single arrays and uses fingerprint-based caching to reduce redundant network requests. The tool covers a broad range of c

RED_HAWK is a penetration testing framework and reconnaissance suite designed for information gathering and vulnerability assessment. It provides a toolkit for infrastructure reconnaissance, technology stack detection, automated web spidering, and security scanning. The project distinguishes itself through a multi-stage reconnaissance pipeline that maps attack surfaces. This includes DNS-based infrastructure mapping to resolve network layouts and pattern-based detection to identify specific content management systems and server stacks. The system covers a broad range of capabilities includin

openai-go is an LLM SDK for Go and a client for interacting with OpenAI services. It provides type-safe bindings to generate text, images, and audio via REST endpoints, enabling the integration of large language models and AI assistant orchestration into Go applications. The library serves as an agent orchestration tool for managing stateful conversation threads and autonomous agents with integrated tool calling and file search. It also functions as an asynchronous batch processing client for monitoring large-scale request groups and fine-tuning jobs, alongside a management SDK for controllin

This is a Python SDK for interacting with large language models via API. It serves as a client library to generate text, process messages, and manage conversational states, while providing a specialized interface for connecting to models hosted across different cloud infrastructure providers. The SDK includes a tool-calling framework that maps Python functions to JSON schemas, allowing models to execute external tools. It also features a built-in token counting utility to estimate input size before transmission and a server-sent events client for receiving model tokens in real time. The libr