awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
·
REMitchell avatar

REMitchell/python-scraping

0
View on GitHub↗
4,714 stars·2,524 forks·Jupyter Notebook·16 views

Python Scraping

This project is a Python web scraping library and automated data collection suite. It provides tools for extracting structured data from websites, implementing web crawlers to navigate site links, and parsing HTML DOM structures to isolate specific elements and attributes.

The toolkit includes a pipeline for processing unstructured text and cleaning raw web content to extract meaningful information. It also features capabilities for image data extraction and the integration of external APIs to retrieve structured data from remote endpoints.

The system covers broad capability areas including automated web data extraction, web crawling workflows, and techniques to bypass scraping obstacles using proxies and CAPTCHA solvers.

Features

  • Web Crawling - Systematically discovers, navigates, and indexes web content across domains for automated data collection.
  • Web Scraping Suites - Provides a complete suite for automated web data collection including proxy management and CAPTCHA solvers.
  • DOM Tree Construction - Implements DOM tree construction to isolate specific elements and attributes from website source code.
  • Scraping and Anti-Detection - Utilizes proxies and CAPTCHA solvers to bypass bot detection and overcome site restrictions.
  • HTTP Content Retrievers - Provides HTTP content retrievers to fetch raw web content as sources for data pipelines.
  • Recursive Web Discovery - Implements recursive link crawling to automatically discover and navigate website hierarchies.
  • Web Data Extraction - Programmatically scrapes and processes web content to capture targeted information using scripts.
  • Headless Browser Automation - Provides headless browser automation to interact with dynamic content and execute JavaScript during collection.
  • Web Scraping Techniques - Implements techniques to bypass access blocks and avoid detection while gathering web data.
  • Proxy Routing - Reroutes network traffic through proxies to hide origin identity and avoid IP-based blocks.
  • HTML Parsers - Converts raw HTML strings into structured tree representations for programmatic analysis and element isolation.
  • Request-Response Loop Automation - Automates the HTTP request-response cycle to fetch raw HTML and JSON content for local processing.
  • Web Crawlers - Implements an automated system that traverses website hierarchies to discover and extract content from multiple pages.
  • Web Data Extractors - Implements utilities for parsing and structuring specific information extracted from web pages.
  • Web Scraping Libraries - Provides a Python library specifically designed for extracting structured data from websites.
  • Image-to-Text Extraction - Includes capabilities to convert image pixels into structured and usable data.
  • Image Processing - Analyzes image data to extract key information and convert pixels into structured data.
  • Document and Unstructured Extraction - Includes automated processes for parsing unstructured web content into machine-readable formats.
  • Image-Based Structured Extraction - Converts image pixels into structured, usable data through specialized processing techniques.
  • Text Cleaning Pipelines - Ships a pipeline for cleaning and standardizing raw web text to remove inconsistencies.
  • Pattern-Based Extraction - Uses regular expressions and string manipulation for pattern-based extraction of structured data from unstructured text.
  • Unstructured Text Processing - Transforms unstructured raw web text into structured formats through a cleaning and language processing pipeline.
  • External API Integrations - Integrates with external APIs to retrieve structured data from remote endpoints.
  • Web Scraping - Code samples for web scraping with Python.

Star history

Star history chart for remitchell/python-scrapingStar history chart for remitchell/python-scraping

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Frequently asked questions

What does remitchell/python-scraping do?

This project is a Python web scraping library and automated data collection suite. It provides tools for extracting structured data from websites, implementing web crawlers to navigate site links, and parsing HTML DOM structures to isolate specific elements and attributes.

What are the main features of remitchell/python-scraping?

The main features of remitchell/python-scraping are: Web Crawling, Web Scraping Suites, DOM Tree Construction, Scraping and Anti-Detection, HTTP Content Retrievers, Recursive Web Discovery, Web Data Extraction, Headless Browser Automation.

What are some open-source alternatives to remitchell/python-scraping?

Open-source alternatives to remitchell/python-scraping include: lorien/web-scraping — This project is a comprehensive resource directory for web data extraction, providing a curated collection of tools… rchipka/node-osmosis — This project is a Node.js web scraping framework designed to automate data extraction through a programmatic workflow… projectdiscovery/katana — Katana is a web crawler and spider designed for security reconnaissance and web application mapping. It functions as a… apify/crawlee — Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction… wistbean/learn_python3_spider — This project is a comprehensive educational guide and framework for building web scrapers using Python. It provides a… kr1s77/python-crawler-tutorial-starts-from-zero — This project is a Python web scraping tutorial and framework designed for building automated data extraction tools and…

Open-source alternatives to Python Scraping

Similar open-source projects, ranked by how many features they share with Python Scraping.
  • lorien/web-scrapinglorien avatar

    lorien/web-scraping

    7,931View on GitHub↗

    This project is a comprehensive resource directory for web data extraction, providing a curated collection of tools and libraries for parsing data, automating browsers, and managing network operations. It serves as a guide for extracting structured information from HTML, XML, JSON, and PDF formats. The toolkit focuses on advanced data collection strategies, including headless browser automation to interact with JavaScript and a suite of network utilities for DNS resolution and WebSocket connections. It specifically covers methods for bypassing bot protections through proxy pool management, us

    Makefile
    View on GitHub↗7,931
  • rchipka/node-osmosisR

    rchipka/node-osmosis

    4,110View on GitHub↗

    This project is a Node.js web scraping framework designed to automate data extraction through a programmatic workflow of requests, parsing, and document interaction. It functions as a headless web crawler, an HTTP request manager, and a DOM parser and extractor. The framework distinguishes itself by combining a JavaScript execution engine to interact with dynamic content and a hybrid selection system that utilizes both CSS and XPath selectors. It includes specialized middleware for proxy rotation and cookie-jar session management to maintain authenticated states and manage automated traffic.

    JavaScript
    View on GitHub↗4,110
  • projectdiscovery/katanaprojectdiscovery avatar

    projectdiscovery/katana

    15,584View on GitHub↗

    Katana is a web crawler and spider designed for security reconnaissance and web application mapping. It functions as a utility for identifying endpoints, forms, and API structures across web targets by combining standard HTTP request traversal with headless browser automation to render dynamic, JavaScript-heavy content. The tool distinguishes itself through its ability to maintain authenticated sessions and handle complex web interactions, such as automated form submission and captcha resolution. It provides granular control over the discovery process, allowing users to define specific crawl

    Goclicrawlergocrawler
    View on GitHub↗15,584
  • apify/crawleeapify avatar

    apify/crawlee

    24,002View on GitHub↗

    Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob

    TypeScriptapifyautomationcrawler
    View on GitHub↗24,002
    • See all 30 alternatives to Python Scraping→