# dotnetcore/dotnetspider

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/dotnetcore-dotnetspider).**

4,137 stars · 1,057 forks · C# · MIT

## Links

- GitHub: https://github.com/dotnetcore/DotnetSpider
- awesome-repositories: https://awesome-repositories.com/repository/dotnetcore-dotnetspider.md

## Topics

`crawler` `cross-platform` `csharp` `distributed` `dotnetcore`

## Description

DotnetSpider is a .NET web crawling framework and C# data extraction tool designed for automated web page discovery and the retrieval of structured data from the internet at scale. It functions as a high-level web scraping library for collecting information from various websites.

The framework provides capabilities for automated web crawling and large-scale data scraping. It enables web content extraction to facilitate the creation of local databases or the analysis of online information through programmatic web automation within the .NET ecosystem.

The system utilizes a pipeline-based data processing model with asynchronous request handling and concurrent worker execution. It features a task-queue based scheduler, modular storage providers, and an interface-driven implementation for custom scraping logic.

## Tags

### Data & Databases

- [Multi-Page Crawling](https://awesome-repositories.com/f/data-databases/multi-page-crawling.md) — Enables automated multi-page crawling to discover and retrieve content across the internet at scale. ([source](https://github.com/dotnetcore/dotnetspider#readme))
- [Concurrent Scraping Workers](https://awesome-repositories.com/f/data-databases/concurrent-scraping-workers.md) — Provides concurrent scraping workers to maximize throughput during large-scale web data extraction.
- [Large Scale Extraction](https://awesome-repositories.com/f/data-databases/large-scale-data-integration-frameworks/large-scale-extraction.md) — Simplifies the collection of large datasets by extracting specific data points from web pages through a structured process.
- [URL Crawl Queues](https://awesome-repositories.com/f/data-databases/url-crawl-queues.md) — Uses URL crawl queues to manage pending pages and schedule parallel processing across workers.
- [Web Data Scraping](https://awesome-repositories.com/f/data-databases/web-data-scraping.md) — Provides tools for extracting structured data points from web pages using automated scripts. ([source](https://github.com/dotnetcore/dotnetspider#readme))
- [Pluggable Storage Engines](https://awesome-repositories.com/f/data-databases/pluggable-storage-engines.md) — Features pluggable storage engines that decouple the scraping engine from the final data destination.

### Content Management & Publishing

- [Web Content Extraction Utilities](https://awesome-repositories.com/f/content-management-publishing/web-content-extraction-utilities.md) — Provides utilities for retrieving structured information from websites to build local databases.

### Development Tools & Productivity

- [Data Extraction Tools](https://awesome-repositories.com/f/development-tools-productivity/data-extraction-tools.md) — Provides a lightweight and efficient C# tool for collecting structured information from the internet.

### Web Development

- [Web Automation Frameworks](https://awesome-repositories.com/f/web-development/web-automation-frameworks.md) — Provides a foundation for building custom programmatic web automation tools within the .NET ecosystem.
- [Web Crawling](https://awesome-repositories.com/f/web-development/web-automation-scraping/web-scraping-automation/web-crawling.md) — Implements a system for systematically discovering and indexing web content across domains for large-scale collection.
- [Web Crawling Frameworks](https://awesome-repositories.com/f/web-development/web-crawling-frameworks.md) — Serves as a comprehensive .NET framework for automating web data extraction, including scheduling and result management.
- [Web Scraping](https://awesome-repositories.com/f/web-development/web-scraping.md) — Acts as a high-level library for extracting structured data points from websites and online sources at scale.
- [Asynchronous Request Handlers](https://awesome-repositories.com/f/web-development/backend-development/request-response-handling/http-request-handling/asynchronous-request-handlers.md) — Implements asynchronous request handlers to maintain high concurrency when fetching multiple web pages.
- [Interface-Driven Implementations](https://awesome-repositories.com/f/web-development/website-crawlers-and-scrapers/custom-scrapers/interface-driven-implementations.md) — Defines a contract for custom scraping logic that the core engine executes during the page lifecycle.

### Software Engineering & Architecture

- [Data Processing Pipelines](https://awesome-repositories.com/f/software-engineering-architecture/data-processing-pipelines.md) — Utilizes data processing pipelines to pass extracted content through discrete stages for filtering and cleaning.

### Part of an Awesome List

- [Application Frameworks](https://awesome-repositories.com/f/awesome-lists/devtools/application-frameworks.md) — High-level web crawling and scraping framework.