awesome-repositories.com
Blog
awesome-repositories.com

Entdecke die besten Open-Source-Repositories mit KI-gestützter Suche.

EntdeckenKuratierte SuchenOpen-Source-AlternativenSelf-hosted SoftwareBlogSitemap
ProjektÜber unsRanking-MethodikPresseMCP-Server
RechtlichesDatenschutzAGB
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
·
zlzforever avatar

zlzforever/DotnetSpider

0
View on GitHub↗
4,136 Stars·1,057 Forks·C#·MIT·3 Aufrufe

DotnetSpider

DotnetSpider is a .NET web crawler framework and programmable tool designed for traversing websites and capturing structured data from web pages. It functions as a distributed crawling engine that enables the automation of web crawling to discover and extract data.

The framework is designed for distributed data extraction, allowing crawling tasks to be spread across multiple servers to process large volumes of web content. This architecture supports high-performance web scraping and enterprise data collection workflows for gathering structured information.

Features

  • Web Crawling - Implements a system for systematically discovering and indexing web content across domains for large-scale data collection.
  • Distributed Web - Uses a distributed scraping architecture to collect high volumes of web data for analysis.
  • Distributed Web Crawling - Scales data extraction by spreading crawling tasks across multiple nodes via coordinated request queues.
  • Distributed Crawling Engines - Implements a scalable architecture for managing large-scale data collection with rate control and memory management.
  • .NET Frameworks - Provides a high-performance web crawler and scraper framework built specifically for the .NET ecosystem.
  • Web Data Collection - Automates the gathering of structured information from websites using custom scraping rules for business intelligence.
  • Web Data Extraction Tools - Offers utilities for scraping, parsing, and structuring information from web pages into usable data formats.
  • High Performance Scraping - Optimizes data collection through high-throughput request management and protocol efficiency for maximum retrieval speed.
  • Dotnet Crawling Frameworks - Cross-platform, lightweight spider for .NET.

Star-Verlauf

Star-Verlauf für zlzforever/dotnetspiderStar-Verlauf für zlzforever/dotnetspider

KI-Suche

Entdecke weitere awesome Repositories

Beschreibe in einfachen Worten, was du brauchst — die KI bewertet tausende kuratierte Open-Source-Projekte nach Relevanz.

Start searching with AI

Open-Source-Alternativen zu DotnetSpider

Ähnliche Open-Source-Projekte, sortiert nach der Anzahl der gemeinsamen Funktionen mit DotnetSpider.
  • apify/crawleeAvatar von apify

    apify/crawlee

    24,002Auf GitHub ansehen↗

    Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob

    TypeScriptapifyautomationcrawler
    Auf GitHub ansehen↗24,002
  • code4craft/webmagicAvatar von code4craft

    code4craft/webmagic

    11,680Auf GitHub ansehen↗

    Webmagic is a Java web crawling framework designed for building scalable automated crawlers to download and process large volumes of web pages. It functions as a distributed web crawler and dynamic content crawler, utilizing an XPath HTML parser to locate and extract specific data points from page structures. The framework distinguishes itself through its ability to handle dynamic content by rendering JavaScript and executing asynchronous requests to extract data from non-static pages. It also allows users to define and execute crawler logic via scripting languages, enabling the update of col

    Javacrawlerframeworkjava
    Auf GitHub ansehen↗11,680
  • boris-code/feapderAvatar von Boris-code

    Boris-code/feapder

    3,709Auf GitHub ansehen↗

    Feapder is a Python web crawling framework designed for building scalable data extraction systems. It features a distributed spider engine and a headless browser renderer to execute JavaScript and extract content from dynamic web pages. The system includes a scalable data deduplicator to filter duplicate URLs and records during large-scale operations. A crawler monitoring system tracks the health of active scraping jobs and triggers alerts when system anomalies occur. The framework provides capabilities for task scheduling, web data extraction, and resilient workflows that allow crawling tas

    Pythoncrawlerfeapderfeaplat
    Auf GitHub ansehen↗3,709
  • bda-research/node-crawlerAvatar von bda-research

    bda-research/node-crawler

    6,785Auf GitHub ansehen↗

    node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It functions as a rate-limited HTTP client and a headless HTML parser, providing the infrastructure to visit large sets of URLs asynchronously while preventing duplicate processing through task deduplication. The project distinguishes itself through a proxy rotation manager that cycles user agents and proxy servers to bypass access restrictions. It utilizes the HTTP/2 protocol to improve request performance and server compatibility during large-scale scraping operations. The syst

    TypeScriptcheeriocrawlerextract-data
    Auf GitHub ansehen↗6,785
Alle 30 Alternativen zu DotnetSpider anzeigen→

Häufig gestellte Fragen

Was macht zlzforever/dotnetspider?

DotnetSpider is a .NET web crawler framework and programmable tool designed for traversing websites and capturing structured data from web pages. It functions as a distributed crawling engine that enables the automation of web crawling to discover and extract data.

Was sind die Hauptfunktionen von zlzforever/dotnetspider?

Die Hauptfunktionen von zlzforever/dotnetspider sind: Web Crawling, Distributed Web, Distributed Web Crawling, Distributed Crawling Engines, .NET Frameworks, Web Data Collection, Web Data Extraction Tools, High Performance Scraping.

Welche Open-Source-Alternativen gibt es zu zlzforever/dotnetspider?

Open-Source-Alternativen zu zlzforever/dotnetspider sind unter anderem: apify/crawlee — Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction… code4craft/webmagic — Webmagic is a Java web crawling framework designed for building scalable automated crawlers to download and process… boris-code/feapder — Feapder is a Python web crawling framework designed for building scalable data extraction systems. It features a… bda-research/node-crawler — node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It… asciimoo/colly — Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages,… lining0806/pythonspidernotes — PythonSpiderNotes is a comprehensive instructional resource and framework for building web crawlers and extracting…