awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
NanmiCoder avatar

NanmiCoder/CrawlerTutorial

0
View on GitHub↗
4,262 stars·471 forks·Python·20 viewsnanmicoder.github.io/CrawlerTutorial↗

CrawlerTutorial

CrawlerTutorial is a comprehensive Python web scraping tutorial and framework designed for extracting data from static and dynamic websites. It functions as a web data extraction pipeline and an HTTP request orchestrator, covering the full lifecycle of scraping applications from initial fetching to final data storage.

The project provides specialized guidance on anti-bot bypass techniques and web API reverse engineering. It includes methods for evading browser detection through identity masking and proxy rotation, as well as techniques for identifying hidden API endpoints by analyzing network traffic and request signatures.

The framework encompasses a broad set of capabilities, including browser automation for JavaScript-heavy pages, automated user authentication via QR codes or SMS, and session persistence management. It also features data preprocessing tools for cleaning raw text, removing duplicate records, and persisting gathered information into flat files or relational databases.

Features

  • Web Data Extraction - Functions as a comprehensive system for programmatically scraping and processing web content from various sources.
  • Web Scraping Tutorials - Provides an extensive educational guide and project-based materials for performing web scraping using Python.
  • HTML Parsing - Provides tools for extracting structured data from HTML using selectors to isolate repeating data blocks.
  • Web Data Pipelines - Implements an automated pipeline for fetching, cleaning, and storing scraped web data into flat files or databases.
  • Web Page Parsing - Extracts specific text, links, and images by analyzing the structure of downloaded web pages.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI
  • Browser Automation Frameworks - Implements a framework for simulating user interactions to extract content from JavaScript-heavy, dynamic webpages.
  • Headless Browser Automation - Automates headless browser engines to render JavaScript and interact with dynamic web content.
  • Browser Automation - Provides programmatic control of browsers to simulate user interactions and manage isolated execution contexts.
  • HTTP Request Execution - Executes synchronous and asynchronous HTTP requests to retrieve data from web servers.
  • Outbound IP Rotation - Distributes requests across a pool of rotating proxy addresses to bypass server-side IP rate limits.
  • Anti-Bot Evasion - Implements browser fingerprint masquerading and proxy rotation to evade sophisticated bot detection systems.
  • Browser Fingerprint Spoofing - Provides tools to spoof browser fingerprints and TLS parameters to evade automated bot detection systems.
  • Browser Automation - Simulates user interactions in headless browsers to extract data from JavaScript-heavy dynamic pages.
  • Concurrent Crawling Engines - Executes multiple network requests simultaneously using asynchronous tasks to accelerate data extraction.
  • JavaScript-Rendered Content Extractors - Includes capabilities to wait for JavaScript execution and ensure dynamic content is fully rendered before extraction.
  • Network Request Interception - Captures API responses and JSON data directly from network traffic to avoid complex DOM parsing.
  • HTTP Header and Cookie Management - Manages HTTP headers and cookies to mimic real browser behavior and maintain request identity.
  • Anti-Detection Automations - Utilizes anti-detection automations to modify browser properties and bypass bot detection mechanisms.
  • Browser Automation - Controls browser processes and isolated contexts to perform complex web actions and manage sessions.
  • Large-Scale Domain Crawlers - Provides a unified framework to manage the full lifecycle of large-scale web scraping applications.
  • Web Scraping Evasion Tools - Provides specialized tools and techniques to mask automation traces and evade bot detection systems.
  • Web Page Retrievers - Downloads HTML content from websites while managing request frequency and concurrency rules.
  • Web Scraping and Extraction - Provides tools for parsing HTML documents to extract structured text and links from static websites.
  • API Reverse Engineering - Offers methods for analyzing network traffic to identify hidden API endpoints and bypass request signatures.
  • Regex Data Extraction - Uses regular expressions to isolate specific values from unstructured text strings.
  • Web Parsing - Retrieves information from web pages using CSS selectors and XPath expressions.
  • Text Cleaning - Cleans raw scraped text by removing HTML tags and fixing encoding for structured analysis.
  • Data Cleaning Pipelines - Implements data cleaning pipelines to transform raw scraped content into usable formats for analysis.
  • Data Processing - Transforms raw crawled data into structured formats suitable for visualization and analysis.
  • Text Preprocessing - Includes tools for cleaning raw scraped text, removing duplicate records, and transforming data into analysis-ready formats.
  • Web Document Parsing - Extracts specific data from HTML and XML documents using specialized parsing libraries.
  • Extracted Data Storage - Saves gathered information into databases or indexes to support searching and analysis.
  • Flat-File Storage - Writes extracted information to simple text files like CSV or JSON using asynchronous I/O.
  • Multi-Format Data Persistence - Saves extracted information across multiple storage types including JSON and CSV flat files.
  • Relational Database Persistence - Persists structured scraped data into relational databases using upsert logic to prevent duplicates.
  • Pagination Pattern Analysis - Includes methods for analyzing URL structures and HTML elements to determine how to navigate paginated results.
  • API-Based Extractions - Retrieves structured data by constructing authenticated HTTP requests to identified API endpoints.
  • Tabular Data Manipulations - Performs tabular data manipulations using data frames to structure and transform extracted information.
  • Token Bucket Implementations - Implements a token bucket algorithm to control request frequency and prevent server overloading.
  • QR Code & Phone Verification Logins - Automates the login process by simulating QR code scanning and polling for authentication status.
  • Phone Verification Code Logins - Handles SMS-based authentication by automating the entry of phone numbers and verification codes.
  • HTTP Request Orchestrators - Ships a system for orchestrating concurrent network requests with integrated proxy rotation and session persistence.
  • HTTP Traffic Inspection - Identifies hidden API endpoints by capturing and analyzing HTTP request and response traffic.
  • Bot Detection Bypass - Avoids IP bans by applying random request delays and capping crawl volume to mimic human behavior.
  • API Endpoint Discovery - Provides techniques for discovering hidden API endpoints by analyzing network traffic and request signatures.
  • Session State Management - Manages local session state by tracking authentication tokens and cookies throughout the extraction process.
  • Anti-Captcha Strategies - Reduces the occurrence of verification walls by managing request frequency and reusing session cookies.
  • Rate Limit Bypasses - Circumvents server-side IP blocks through identity randomization and request frequency management.
  • Session Authentication Strategies - Implements session authentication strategies including QR-based authorization to access restricted data.
  • Session-Cookie Persistences - Persists and reuses session cookies and tokens to maintain authenticated states across scraping runs.
  • Authentication Token Extraction - Retrieves authentication tokens and session cookies from browser contexts for use in programmatic requests.
  • Protection Bypassers - Implements signature algorithms and header management to circumvent automated API security challenges.
  • User Authentications - Handles user identity verification and session persistence through the management of cookies and CAPTCHA resolution.
  • Concurrent Task Execution - Implements concurrent execution of network requests to improve the throughput of data extraction.
  • Retry Policies - Handles network instability and transient errors by automatically retrying failed scraping operations.
  • Request Rate Limiting - Controls outgoing request frequency using a token bucket mechanism to avoid server overloading.
  • Crawl Request Deduplications - Prevents redundant crawling by filtering and deduplicating extracted URLs using a tracking system.
  • Browser Session Managers - Reuses existing browser sessions to minimize resource overhead and improve execution efficiency.
  • Resource Blocking - Blocks non-essential resources like images and fonts to reduce bandwidth and increase extraction speed.
  • Crawler Identity Masking - Configures HTTP headers and TLS fingerprints to simulate real browser behavior and evade detection.
  • Request Proxying - Hides the origin IP address by routing web requests through a pool of intermediate proxy servers.

    • Star history

    Star history chart for nanmicoder/crawlertutorialStar history chart for nanmicoder/crawlertutorial

    How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

    Frequently asked questions

    What does nanmicoder/crawlertutorial do?

    CrawlerTutorial is a comprehensive Python web scraping tutorial and framework designed for extracting data from static and dynamic websites. It functions as a web data extraction pipeline and an HTTP request orchestrator, covering the full lifecycle of scraping applications from initial fetching to final data storage.

    What are the main features of nanmicoder/crawlertutorial?

    The main features of nanmicoder/crawlertutorial are: Web Data Extraction, Web Scraping Tutorials, HTML Parsing, Web Data Pipelines, Web Page Parsing, Browser Automation Frameworks, Headless Browser Automation, Browser Automation.

    Which projects share features with nanmicoder/crawlertutorial?

    Projects with overlapping indexed features include: apify/crawlee — Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction… wistbean/learn_python3_spider — This project is a comprehensive educational guide and framework for building web scrapers using Python. It provides a… browser-act/skills — This project provides an agentic web interaction engine designed to facilitate autonomous browser automation and… apify/crawlee-python — Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive… guyungy/damaihelper — Damaihelper is a ticketing automation bot and browser automation framework designed to monitor ticket availability and… kr1s77/python-crawler-tutorial-starts-from-zero — This project is a Python web scraping tutorial and framework designed for building automated data extraction tools and…

    Projects sharing features with CrawlerTutorial

    These projects share indexed features with CrawlerTutorial. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
    • apify/crawleeapify avatar

      apify/crawlee

      24,002View on GitHub↗

      Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob

      TypeScriptapifyautomationcrawler
      View on GitHub↗24,002
    • wistbean/learn_python3_spiderwistbean avatar

      wistbean/learn_python3_spider

      21,802View on GitHub↗

      This project is a comprehensive educational guide and framework for building web scrapers using Python. It provides a course-based approach to data extraction, combining a Python crawler framework with tutorials on web reverse engineering and network traffic analysis. The project distinguishes itself by covering advanced extraction challenges, including the decryption of obfuscated JavaScript and the bypass of anti-scraping measures. It specifically addresses mobile application scraping through the simulation of user interactions and the interception of network traffic. The capability surfac

      Pythonpython-scriptpython-spiderpython3
      View on GitHub↗21,802
    • browser-act/skillsbrowser-act avatar

      browser-act/skills

      2,554View on GitHub↗

      This project provides an agentic web interaction engine designed to facilitate autonomous browser automation and large-scale data extraction. It serves as a framework for building and deploying agents that can navigate complex, JavaScript-rendered websites, interact with page elements, and execute multi-step workflows. By providing a structured environment for browser control, the system enables the creation of reusable automation scripts that can be deployed across diverse web platforms. The platform distinguishes itself through a comprehensive suite of security and traffic management tools,

      Pythonai-agentsautomationclaude-cli
      View on GitHub↗2,554
    • apify/crawlee-pythonapify avatar

      apify/crawlee-python

      8,097View on GitHub↗

      Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

      Pythonapifyautomationbeautifulsoup
      View on GitHub↗8,097
    Compare all 30 related projects→