Awesome GitHub RepositoriesGo Crawling Frameworks

Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages, following links, and extracting structured data from HTML and XML documents. The framework includes a distributed scraping engine designed to spread data collection tasks across multiple instances to increase throughput. It ensures compliance with website owner policies by automatically reading and respecting robots.txt files. The system manages request lifecycles through domain-based rate limiting, concurrency controls, and session management via a stateful cookie jar. It s

Pholcus 是一个用 Go 编写的分布式网络爬虫框架，专为高并发数据提取而设计。它作为一个分布式爬虫编排器和动态数据提取引擎，利用服务器-客户端架构在多个节点间协调任务。 该系统集成了无头浏览器引擎来渲染动态内容并执行 JavaScript，从而能够从单页应用 (SPA) 中提取数据。它具有用于配置爬虫参数和监控执行进度的 Web 管理界面，并支持通过热重载配置文件更新提取规则，无需重启系统。 流量管理通过代理池轮换和请求随机化来处理，以规避机器人检测并避免速率限制。该框架包括基于状态的检查点恢复功能，可在故障后恢复任务，并提供可插拔的存储适配器，将提取的数据导出到数据库、消息队列或文件中。

Hakrawler is a command-line web spider tool designed for security reconnaissance, built to crawl target websites and extract hyperlinks along with JavaScript file references. As a focused reconnaissance utility, it collects every discoverable URL and script source from a given domain, mapping the attack surface for penetration testing and vulnerability assessment. The tool differentiates itself through its concurrent architecture: a fixed-size goroutine pool fetches pages in parallel, while CSS selectors parse HTML to extract anchor and script references. A depth-aware recursion limiter preve

See the video on the Youtube.

gocrawl is a polite, slim and concurrent web crawler written in Go.

A simple, higher level interface for Go web scraping.

Package fetchbot provides a simple and flexible web crawler that follows the robots.txt policies and crawl delays.

Dataflow kit ("DFK") is a Web Scraping framework for Gophers. It extracts data from web pages, following the specified CSS Selectors.

open source, restful, distributed crawler engine

Creeper is a next-generation crawler which fetches web page by creeper script. As a cross-platform embedded crawler, you can use it for your news app, subscribe program, etc.