awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
dotnetcore avatar

dotnetcore/DotnetSpider

0
View on GitHub↗

DotnetSpider

DotnetSpider is a .NET web crawling framework and C# data extraction tool designed for automated web page discovery and the retrieval of structured data from the internet at scale. It functions as a high-level web scraping library for collecting information from various websites.

The framework provides capabilities for automated web crawling and large-scale data scraping. It enables web content extraction to facilitate the creation of local databases or the analysis of online information through programmatic web automation within the .NET ecosystem.

The system utilizes a pipeline-based data processing model with asynchronous request handling and concurrent worker execution. It features a task-queue based scheduler, modular storage providers, and an interface-driven implementation for custom scraping logic.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Features

  • Multi-Page Crawling - Enables automated multi-page crawling to discover and retrieve content across the internet at scale.
  • Web Content Extraction Utilities - Provides utilities for retrieving structured information from websites to build local databases.
  • Concurrent Scraping Workers - Provides concurrent scraping workers to maximize throughput during large-scale web data extraction.
  • Large Scale Extraction - Simplifies the collection of large datasets by extracting specific data points from web pages through a structured process.
  • URL Crawl Queues - Uses URL crawl queues to manage pending pages and schedule parallel processing across workers.
  • Web Data Scraping - Provides tools for extracting structured data points from web pages using automated scripts.
  • Data Extraction Tools - Provides a lightweight and efficient C# tool for collecting structured information from the internet.
  • Web Automation Frameworks - Provides a foundation for building custom programmatic web automation tools within the .NET ecosystem.
  • Web Crawling - Implements a system for systematically discovering and indexing web content across domains for large-scale collection.
  • Web Crawling Frameworks - Serves as a comprehensive .NET framework for automating web data extraction, including scheduling and result management.
  • Web Scraping - Acts as a high-level library for extracting structured data points from websites and online sources at scale.
  • Pluggable Storage Engines - Features pluggable storage engines that decouple the scraping engine from the final data destination.
  • Data Processing Pipelines - Utilizes data processing pipelines to pass extracted content through discrete stages for filtering and cleaning.
  • Asynchronous Request Handlers - Implements asynchronous request handlers to maintain high concurrency when fetching multiple web pages.
  • Interface-Driven Implementations - Defines a contract for custom scraping logic that the core engine executes during the page lifecycle.
  • Application Frameworks - High-level web crawling and scraping framework.
4,137 stars·1,057 forks·C#·MIT·22 views

Star history

Star history chart for dotnetcore/dotnetspiderStar history chart for dotnetcore/dotnetspider

How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

Frequently asked questions

What does dotnetcore/dotnetspider do?

DotnetSpider is a .NET web crawling framework and C# data extraction tool designed for automated web page discovery and the retrieval of structured data from the internet at scale. It functions as a high-level web scraping library for collecting information from various websites.

What are the main features of dotnetcore/dotnetspider?

The main features of dotnetcore/dotnetspider are: Multi-Page Crawling, Web Content Extraction Utilities, Concurrent Scraping Workers, Large Scale Extraction, URL Crawl Queues, Web Data Scraping, Data Extraction Tools, Web Automation Frameworks.

Which projects share features with dotnetcore/dotnetspider?

Projects with overlapping indexed features include: code4craft/webmagic — Webmagic is a Java web crawling framework designed for building scalable automated crawlers to download and process… apify/crawlee-python — Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive… kr1s77/awesome-python-login-model — This project is a Python-based automation toolkit designed to manage programmatic authentication and session… asciimoo/colly — Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages,… any4ai/anycrawl — AnyCrawl is an AI-powered data extractor, automated web crawler, and headless browser orchestrator. It serves as a web… binux/pyspider — PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for…

Projects sharing features with DotnetSpider

These projects share indexed features with DotnetSpider. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
  • code4craft/webmagiccode4craft avatar

    code4craft/webmagic

    11,680View on GitHub↗

    Webmagic is a Java web crawling framework designed for building scalable automated crawlers to download and process large volumes of web pages. It functions as a distributed web crawler and dynamic content crawler, utilizing an XPath HTML parser to locate and extract specific data points from page structures. The framework distinguishes itself through its ability to handle dynamic content by rendering JavaScript and executing asynchronous requests to extract data from non-static pages. It also allows users to define and execute crawler logic via scripting languages, enabling the update of col

    Javacrawlerframeworkjava
    View on GitHub↗11,680
  • apify/crawlee-pythonapify avatar

    apify/crawlee-python

    8,097View on GitHub↗

    Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

    Pythonapifyautomationbeautifulsoup
    View on GitHub↗8,097
  • kr1s77/awesome-python-login-modelKr1s77 avatar

    Kr1s77/awesome-python-login-model

    16,225View on GitHub↗

    This project is a Python-based automation toolkit designed to manage programmatic authentication and session persistence across web services. It provides a framework for executing automated login sequences, including the handling of interactive security challenges such as QR code verification and captcha resolution. The toolkit distinguishes itself by simulating native mobile application environments, allowing for the execution of scripts that require specific device-level headers and behaviors. It also incorporates hook-based interception to monitor workflow states and manage exceptions duri

    Python163mail-loginbilibili-logindouban-spider
    View on GitHub↗16,225
  • asciimoo/collyasciimoo avatar

    asciimoo/colly

    25,348View on GitHub↗

    Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages, following links, and extracting structured data from HTML and XML documents. The framework includes a distributed scraping engine designed to spread data collection tasks across multiple instances to increase throughput. It ensures compliance with website owner policies by automatically reading and respecting robots.txt files. The system manages request lifecycles through domain-based rate limiting, concurrency controls, and session management via a stateful cookie jar. It s

    Go
    View on GitHub↗25,348
Compare all 30 related projects→