Colly

Colly is a high-performance web scraping framework designed for the automated extraction of structured data from websites. It provides a programmable toolkit that manages the complexities of large-scale data collection, including concurrent request orchestration, automatic cookie handling, and robots.txt compliance. By utilizing an asynchronous execution model, the engine maintains high throughput while preventing resource exhaustion during recursive or distributed crawling tasks.

The framework is distinguished by its modular, event-driven architecture, which allows developers to hook into specific lifecycle stages of a network request to process content or control flow. It features a flexible middleware pipeline for handling proxy rotation, user agents, and rate limiting, alongside an interface-driven storage layer that supports swapping default in-memory state for persistent external databases. This design enables the coordination of multiple scraping instances and the maintenance of crawl history across application restarts.

Beyond its core engine, the project offers extensive customization options for network transport, including support for custom round-trippers to manage connection pooling and timeouts. It also provides robust observability tools, allowing for the attachment of custom debuggers and logging observers to monitor internal state during execution. Developers can further extend functionality through a plugin system or by sharing request context and configuration across different collector instances to support complex, multi-stage data extraction workflows.

Features

Web Scraping Engines - Extracts web content using a high-performance engine that manages concurrency, caching, and robots.txt compliance.

Web Scraping Frameworks - Provides a programmable toolkit for extracting structured data through automated request handling and parsing workflows.

Concurrent Crawling Engines - Manages high-performance asynchronous network requests and distributed state across parallel scraping tasks.

Web Data Extractors - Automates the retrieval and parsing of structured information from websites to build datasets.

High-Volume Web Scrapers - Manages large-scale data collection tasks that require proxy rotation, rate limiting, and persistent state.

Web Crawling Orchestrators - Manages asynchronous network execution to maintain high throughput during large-scale data extraction.

Asynchronous Request Runners - Performs network requests in the background to keep the application responsive during large-scale extraction.

Distributed Crawler Orchestrators - Coordinates multiple scraping instances across different environments by sharing state and configuration.

Event-Driven Data Extractors - Executes user-defined functions at specific lifecycle stages to process content and control network flow.

Request Middleware Pipelines - Processes outgoing requests through a chain of configurable components to handle proxy rotation and rate limiting.

Web Crawlers - Elegant and fast web scraping framework.

Proxy Rotation Services - Distributes network traffic across multiple proxy servers to avoid IP-based blocking during data collection.

Extensible Scraping Frameworks - Provides a modular framework that allows developers to inject custom storage, proxy logic, and processing callbacks.

Event-Driven Scraping Hooks - Executes user-defined functions at specific lifecycle stages of a network request to process data.

Distributed State Persistence - Maintains shared cookie and URL history state across multiple independent instances in distributed environments.

Storage Abstraction Layers - Decouples state management from the core engine to allow swapping in-memory storage for persistent databases.

HTTP Transport Configurations - Customizes the networking layer to manage proxies, timeouts, and connection pooling for outgoing requests.

Collector Lifecycle Managers - Creates a collector object to manage network communication and trigger registered callback functions.

Persistent Application State - Maintains cookies and visited URL history across application restarts and long-running data extraction sessions.

Persistent Storage Backends - Swaps in-memory state management for persistent databases to maintain crawl history.

Storage Backend Adapters - Overrides default memory storage by assigning custom backends to manage persistent data.

Crawler Extensions - Provides pluggable extensions to automate user agent rotation and referrer management during web data collection.

Request Context Propagation - Carries metadata and state across asynchronous network calls by embedding values within the request object.

Scraping Callback Registrars - Attaches callback functions to monitor request lifecycles and process retrieved content.

gocollycolly

Features

Star history