Awesome GitHub RepositoriesData Extraction Pipelines

This project is a community-maintained, open-source job aggregator that provides a curated database of internship opportunities. It centralizes scattered professional listings into a structured, searchable collection categorized by industry, role, and location to assist students in their career search. The repository distinguishes itself by utilizing a version-controlled data store, where all job listings are maintained as plain text files. This approach enables transparent history tracking and granular change analysis through standard diffing tools. The project relies on an automated data ex

Maigret is an open-source intelligence framework designed for automated digital footprint discovery and identity investigation. It functions as a search engine that aggregates profile metadata by querying thousands of websites for specific usernames, mapping an individual's online presence across diverse platforms. The tool distinguishes itself through recursive discovery capabilities, which identify links within discovered profiles to expand the scope of an investigation automatically. It supports cross-platform identity correlation by mapping disparate accounts and pseudonymous personas, in

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Maxun is an open-source web scraping and automation platform designed to transform dynamic website content into structured data. By leveraging artificial intelligence to interpret natural language prompts, the system identifies page elements and extracts information without requiring manual selector configuration. It serves as a bridge between raw web content and intelligent workflows, providing structured outputs in formats optimized for large language model ingestion and agent-based applications. The platform distinguishes itself through its ability to handle complex, authenticated, and dyn

Binwalk is a firmware analysis and reverse engineering tool designed to identify, extract, and analyze embedded files and data types within binary firmware blobs. It functions as a binary file signature scanner and entropy analyzer to decompose firmware images into their constituent components. The tool distinguishes itself by combining signature-based pattern matching for known binary headers with entropy analysis. By calculating data randomness across file offsets, it can locate compressed or encrypted sections that do not possess known signatures. The project covers binary data forensics

Spider-flow is a Java-based web crawling and data extraction platform that provides a centralized environment for managing automated information gathering. It functions as a no-code tool, allowing users to define complex data collection pipelines through a visual, drag-and-drop interface rather than manual programming. The platform distinguishes itself through a graph-based workflow orchestration system where users link discrete nodes to define navigation and parsing logic. It supports dynamic content crawling by integrating headless browsers to execute JavaScript and render page content that

weiboSpider is a Python web scraper and social media crawler designed to extract user profiles, posts, and engagement metrics from Sina Weibo. It functions as an automated data pipeline for academic research and trend analysis, collecting long-form text and multimedia content. The tool distinguishes itself through the use of browser session cookies to authenticate requests and access protected profiles. It implements randomized request pacing and global pauses to manage traffic and avoid platform rate limits, while supporting incremental crawling to capture only new content based on timestamp

Tabula is a PDF table extraction tool and data scraper designed to isolate tabular structures within text-based PDF files. It functions as a converter that transforms these layouts into structured CSV or spreadsheet formats for data recovery and analysis. The project provides both a visual interface for manually selecting table areas and a headless command-line interface. This dual approach allows for a choice between manual data recovery via visual-area selection and the integration of table extraction into automated data pipelines. The extraction process utilizes Java-based PDF parsing and

dlt est un outil d'ingestion de données Python et un framework de pipeline ETL conçu pour récupérer des données depuis diverses sources et les persister dans des destinations structurées. Il fonctionne comme un moteur d'inférence de schéma qui détecte automatiquement les types de données et aplatit les structures JSON imbriquées en tables relationnelles, déplaçant les données des sources vers des lakehouses, des entrepôts ou des bases de données vectorielles. Le projet se distingue par une génération de pipeline alimentée par l'IA, utilisant de grands modèles de langage pour échafauder le code d'extraction et les connecteurs pour les API REST. Il prend également en charge le stockage vectoriel multimodal et la population spécialisée de bases de données vectorielles pour prendre en charge les applications d'IA et de machine learning. Le framework couvre un large éventail de capacités, incluant l'évolution automatique du schéma, le chargement incrémentiel de données via le suivi d'état et la validation de la qualité des données par l'application de contrats de données. Il fournit des outils pour la normalisation des données relationnelles, les transformations pré- et post-chargement, et une variété d'adaptateurs de destination pour les bases de données SQL et les magasins d'objets cloud. L'observabilité est gérée via des tableaux de bord d'exécution de pipeline, le suivi de lignage des colonnes et la vérification de version de schéma utilisant des hachages basés sur le contenu.

Botasaurus is a Python web scraping framework and headless browser automation system used to build scalable data extraction tools. It functions as a web data extraction tool and OCR document parser, converting website content, images, and PDF files into structured formats such as JSON, CSV, and Excel. The framework distinguishes itself by providing a scraper management interface that allows Python functions to be wrapped in a web-based UI or deployed as standalone desktop applications. This enables non-technical users to trigger extraction jobs and manage tasks via a graphical interface or RE

This project is an LLM-powered web crawler and data extractor that uses large language models to navigate websites and parse content into structured JSON or Markdown formats. It functions as an automated browser orchestrator and domain discovery engine, interpreting plain English instructions to identify relevant pages and extract specific information. The system distinguishes itself through agentic browser automation, allowing it to perform human-like interactions such as clicking buttons and scrolling based on natural language commands. It employs goal-oriented crawling to analyze website s

Roadmap-Docs is a technical career roadmap repository that provides structured learning paths for software engineering, data science, and artificial intelligence roles. It functions as a professional development curriculum, mapping essential technical milestones and industry-standard tools to guide career advancement. The platform integrates a job market trend analyzer that evaluates industry job postings to identify high-demand skills and competencies. By utilizing an automated data pipeline, the system updates these educational roadmaps to reflect real-time industry requirements and evolvin

Spider est une plateforme basée sur le web conçue pour l'extraction automatisée de données, fournissant un framework centralisé pour collecter, traiter et router des informations structurées à partir de sites web. Elle fonctionne comme un pipeline complet qui gère l'ensemble du cycle de vie de la collecte de données, de la configuration initiale au stockage final dans des bases de données externes ou des files d'attente de messages. La plateforme se distingue par une interface de configuration visuelle qui permet aux utilisateurs de définir des règles d'extraction et de gérer des modèles de scraping sans écrire de code personnalisé. Elle prend en charge la récupération de contenu statique et dynamique en intégrant l'automatisation de navigateur headless, qui exécute du JavaScript côté client pour capturer des données à partir de sites web modernes et interactifs. Les utilisateurs peuvent également utiliser l'extraction heuristique pour extraire automatiquement le texte principal des articles et les métadonnées sans configuration manuelle de sélecteurs. Au-delà de la simple collecte, le système inclut des outils pour l'analyse de contenu structuré et le mappage de relations. Il identifie les mots-clés, détecte les entités et visualise les connexions entre les personnes, les lieux et les articles pour fournir un contexte plus profond aux informations recueillies. La plateforme dispose également de capacités intégrées de recherche et d'indexation, permettant aux utilisateurs d'interroger les enregistrements stockés et de gérer le flux de données via un tableau de bord unifié.