Awesome GitHub RepositoriesData Extraction Pipelines

This project is a community-maintained, open-source job aggregator that provides a curated database of internship opportunities. It centralizes scattered professional listings into a structured, searchable collection categorized by industry, role, and location to assist students in their career search. The repository distinguishes itself by utilizing a version-controlled data store, where all job listings are maintained as plain text files. This approach enables transparent history tracking and granular change analysis through standard diffing tools. The project relies on an automated data ex

Maigret is an open-source intelligence framework designed for automated digital footprint discovery and identity investigation. It functions as a search engine that aggregates profile metadata by querying thousands of websites for specific usernames, mapping an individual's online presence across diverse platforms. The tool distinguishes itself through recursive discovery capabilities, which identify links within discovered profiles to expand the scope of an investigation automatically. It supports cross-platform identity correlation by mapping disparate accounts and pseudonymous personas, in

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Maxun is an open-source web scraping and automation platform designed to transform dynamic website content into structured data. By leveraging artificial intelligence to interpret natural language prompts, the system identifies page elements and extracts information without requiring manual selector configuration. It serves as a bridge between raw web content and intelligent workflows, providing structured outputs in formats optimized for large language model ingestion and agent-based applications. The platform distinguishes itself through its ability to handle complex, authenticated, and dyn

Binwalk is a firmware analysis and reverse engineering tool designed to identify, extract, and analyze embedded files and data types within binary firmware blobs. It functions as a binary file signature scanner and entropy analyzer to decompose firmware images into their constituent components. The tool distinguishes itself by combining signature-based pattern matching for known binary headers with entropy analysis. By calculating data randomness across file offsets, it can locate compressed or encrypted sections that do not possess known signatures. The project covers binary data forensics

Spider-flow is a Java-based web crawling and data extraction platform that provides a centralized environment for managing automated information gathering. It functions as a no-code tool, allowing users to define complex data collection pipelines through a visual, drag-and-drop interface rather than manual programming. The platform distinguishes itself through a graph-based workflow orchestration system where users link discrete nodes to define navigation and parsing logic. It supports dynamic content crawling by integrating headless browsers to execute JavaScript and render page content that

weiboSpider is a Python web scraper and social media crawler designed to extract user profiles, posts, and engagement metrics from Sina Weibo. It functions as an automated data pipeline for academic research and trend analysis, collecting long-form text and multimedia content. The tool distinguishes itself through the use of browser session cookies to authenticate requests and access protected profiles. It implements randomized request pacing and global pauses to manage traffic and avoid platform rate limits, while supporting incremental crawling to capture only new content based on timestamp

Tabula is a PDF table extraction tool and data scraper designed to isolate tabular structures within text-based PDF files. It functions as a converter that transforms these layouts into structured CSV or spreadsheet formats for data recovery and analysis. The project provides both a visual interface for manually selecting table areas and a headless command-line interface. This dual approach allows for a choice between manual data recovery via visual-area selection and the integration of table extraction into automated data pipelines. The extraction process utilizes Java-based PDF parsing and

dlt ist ein Python-Tool zur Datenaufnahme und ein ETL-Pipeline-Framework, das darauf ausgelegt ist, Daten aus verschiedenen Quellen abzurufen und in strukturierten Zielen zu speichern. Es fungiert als Schema-Inferenz-Engine, die automatisch Datentypen erkennt und verschachtelte JSON-Strukturen in relationale Tabellen flacht, wobei Daten von Quellen in Lakehouses, Warehouses oder Vektordatenbanken verschoben werden. Das Projekt zeichnet sich durch KI-gestützte Pipeline-Generierung aus, die Large Language Models nutzt, um Extraktionscode und Konnektoren für REST-APIs zu erstellen. Es unterstützt zudem multimodale Vektorspeicherung und die spezialisierte Befüllung von Vektordatenbanken zur Unterstützung von KI- und Machine-Learning-Anwendungen. Das Framework deckt ein breites Spektrum an Funktionen ab, einschließlich automatisierter Schema-Evolution, inkrementellem Datenladen mittels Statusverfolgung und Datenqualitätsvalidierung durch die Durchsetzung von Datenverträgen. Es bietet Tools für relationale Datennormalisierung, Pre- und Post-Load-Transformationen sowie eine Vielzahl von Ziel-Adaptern für SQL-Datenbanken und Cloud-Objektspeicher. Die Observability wird durch Pipeline-Ausführungs-Dashboards, Spalten-Lineage-Tracking und Schema-Versionsverifizierung mittels inhaltsbasierter Hashes gehandhabt.

Botasaurus is a Python web scraping framework and headless browser automation system used to build scalable data extraction tools. It functions as a web data extraction tool and OCR document parser, converting website content, images, and PDF files into structured formats such as JSON, CSV, and Excel. The framework distinguishes itself by providing a scraper management interface that allows Python functions to be wrapped in a web-based UI or deployed as standalone desktop applications. This enables non-technical users to trigger extraction jobs and manage tasks via a graphical interface or RE

This project is an LLM-powered web crawler and data extractor that uses large language models to navigate websites and parse content into structured JSON or Markdown formats. It functions as an automated browser orchestrator and domain discovery engine, interpreting plain English instructions to identify relevant pages and extract specific information. The system distinguishes itself through agentic browser automation, allowing it to perform human-like interactions such as clicking buttons and scrolling based on natural language commands. It employs goal-oriented crawling to analyze website s

Roadmap-Docs is a technical career roadmap repository that provides structured learning paths for software engineering, data science, and artificial intelligence roles. It functions as a professional development curriculum, mapping essential technical milestones and industry-standard tools to guide career advancement. The platform integrates a job market trend analyzer that evaluates industry job postings to identify high-demand skills and competencies. By utilizing an automated data pipeline, the system updates these educational roadmaps to reflect real-time industry requirements and evolvin

Spider ist eine webbasierte Plattform zur automatisierten Datenextraktion, die ein zentralisiertes Framework zur Sammlung, Verarbeitung und Weiterleitung strukturierter Informationen von Websites bietet. Sie fungiert als umfassende Pipeline, die den gesamten Lebenszyklus der Datengewinnung verwaltet, von der anfänglichen Konfiguration bis zur finalen Speicherung in externen Datenbanken oder Message-Queues. Die Plattform zeichnet sich durch ein visuelles Konfigurations-Interface aus, das es Benutzern erlaubt, Extraktionsregeln zu definieren und Scraping-Templates zu verwalten, ohne benutzerdefinierten Code schreiben zu müssen. Sie unterstützt sowohl statische als auch dynamische Inhaltsabrufe durch die Integration von Headless-Browser-Automatisierung, die clientseitiges JavaScript ausführt, um Daten von modernen, interaktiven Websites zu erfassen. Benutzer können zudem heuristische Extraktion nutzen, um Kernartikeltexte und Metadaten automatisch ohne manuelle Selektorkonfiguration abzurufen. Über die einfache Sammlung hinaus enthält das System Tools zur strukturierten Inhaltsanalyse und Beziehungs-Mapping. Es identifiziert Schlüsselwörter, erkennt Entitäten und visualisiert Verbindungen zwischen Personen, Orten und Artikeln, um tiefere Kontexte für gesammelte Informationen bereitzustellen. Die Plattform bietet zudem integrierte Such- und Indexierungsfunktionen, die es Benutzern erlauben, gespeicherte Datensätze abzufragen und den Datenfluss über ein einheitliches Dashboard zu verwalten.