asciimoo/colly
Colly
Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages, following links, and extracting structured data from HTML and XML documents.
The framework includes a distributed scraping engine designed to spread data collection tasks across multiple instances to increase throughput. It ensures compliance with website owner policies by automatically reading and respecting robots.txt files.
The system manages request lifecycles through domain-based rate limiting, concurrency controls, and session management via a stateful cookie jar. It supports large-scale data mining through parallel workflow execution and pluggable response caching to reduce redundant network requests.
Features
- Web Crawling - Provides a system to systematically discover, navigate, and index web content across multiple domains.
- Web Content Extractors - Parses web responses to isolate and extract specific data points using defined rules.
- Automated Web Scraping - Automates the extraction of structured information from HTML and XML websites using Go.
- Multi-Page Crawling - Automatically follows links and traverses multiple web pages to discover and extract structured data.
- Structured Data Extraction - Parses HTML content to collect specific, structured data points for mining and archiving.
- Recursive Discovery Engines - Automatically discovers and queues new URLs from parsed HTML to traverse entire website structures recursively.
- Web Scraping - Extracts structured information from HTML and XML documents using defined rules.
- Session & Cookie Handlers - Maintains a stateful cookie jar to persist session data and authentication across multiple HTTP requests.
- Robots Policy Enforcers - Automatically reads and respects robots.txt files to ensure crawling compliance with website owner policies.
- Concurrent Request Limits - Implements a semaphore-based system to limit the number of simultaneous active network requests and prevent server overloading.
- Request Rate Limiting - Enforces domain-based rate limiting and request delays to prevent overloading target servers.
- Domain-Based Rate Limiting - Enforces minimum delays between requests to the same host to ensure compliant and stable crawling.
- Robots Exclusion Compliance - Automatically reads and respects robots.txt files to ensure crawling activities comply with website owner policies.
- Session State Management - Manages cookies and session states to interact with authenticated or state-dependent web pages.
- Web Crawlers - Functions as a concurrent system that traverses websites to discover and extract content from multiple pages.
- Distributed Web - Distributes scraping tasks across multiple instances to increase the volume and throughput of collected web data.
- Web Page Caches - Provides pluggable response caching to reduce redundant network requests and increase scraping speed.
- Task Distribution Scaling - Distributes scraping workloads across multiple workers to increase total data collection capacity.
- Task Coordinations - Synchronizes scraping workloads across distributed nodes using an external messaging or storage backend.
- Scraping Architectures - Implements a scalable architecture for spreading data collection tasks across distributed server networks.
- Lifecycle Event Callbacks - Provides callbacks that trigger at specific stages of the request lifecycle, such as when a page start or response is received.
- Parallel Task Execution - Implements asynchronous and parallel execution modes to optimize the speed of data collection.
- Response Caching - Includes a pluggable response caching system to reduce redundant network requests and increase throughput.
- Go Crawling Frameworks - Fast and elegant scraping framework.
- Text Processing - High-performance web scraping framework.
Star history
How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
