Awesome GitHub RepositoriesData Extraction Pipelines

This project is a community-maintained, open-source job aggregator that provides a curated database of internship opportunities. It centralizes scattered professional listings into a structured, searchable collection categorized by industry, role, and location to assist students in their career search. The repository distinguishes itself by utilizing a version-controlled data store, where all job listings are maintained as plain text files. This approach enables transparent history tracking and granular change analysis through standard diffing tools. The project relies on an automated data ex

Maigret is an open-source intelligence framework designed for automated digital footprint discovery and identity investigation. It functions as a search engine that aggregates profile metadata by querying thousands of websites for specific usernames, mapping an individual's online presence across diverse platforms. The tool distinguishes itself through recursive discovery capabilities, which identify links within discovered profiles to expand the scope of an investigation automatically. It supports cross-platform identity correlation by mapping disparate accounts and pseudonymous personas, in

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Maxun is an open-source web scraping and automation platform designed to transform dynamic website content into structured data. By leveraging artificial intelligence to interpret natural language prompts, the system identifies page elements and extracts information without requiring manual selector configuration. It serves as a bridge between raw web content and intelligent workflows, providing structured outputs in formats optimized for large language model ingestion and agent-based applications. The platform distinguishes itself through its ability to handle complex, authenticated, and dyn

Binwalk is a firmware analysis and reverse engineering tool designed to identify, extract, and analyze embedded files and data types within binary firmware blobs. It functions as a binary file signature scanner and entropy analyzer to decompose firmware images into their constituent components. The tool distinguishes itself by combining signature-based pattern matching for known binary headers with entropy analysis. By calculating data randomness across file offsets, it can locate compressed or encrypted sections that do not possess known signatures. The project covers binary data forensics

Spider-flow is a Java-based web crawling and data extraction platform that provides a centralized environment for managing automated information gathering. It functions as a no-code tool, allowing users to define complex data collection pipelines through a visual, drag-and-drop interface rather than manual programming. The platform distinguishes itself through a graph-based workflow orchestration system where users link discrete nodes to define navigation and parsing logic. It supports dynamic content crawling by integrating headless browsers to execute JavaScript and render page content that

weiboSpider is a Python web scraper and social media crawler designed to extract user profiles, posts, and engagement metrics from Sina Weibo. It functions as an automated data pipeline for academic research and trend analysis, collecting long-form text and multimedia content. The tool distinguishes itself through the use of browser session cookies to authenticate requests and access protected profiles. It implements randomized request pacing and global pauses to manage traffic and avoid platform rate limits, while supporting incremental crawling to capture only new content based on timestamp

Tabula is a PDF table extraction tool and data scraper designed to isolate tabular structures within text-based PDF files. It functions as a converter that transforms these layouts into structured CSV or spreadsheet formats for data recovery and analysis. The project provides both a visual interface for manually selecting table areas and a headless command-line interface. This dual approach allows for a choice between manual data recovery via visual-area selection and the integration of table extraction into automated data pipelines. The extraction process utilizes Java-based PDF parsing and

dlt este un instrument de ingestie a datelor Python și un framework de pipeline ETL conceput pentru a prelua date din surse diverse și a le persista în destinații structurate. Funcționează ca un motor de inferență a schemei care detectează automat tipurile de date și aplatizează structurile JSON imbricate în tabele relaționale, mutând datele din surse către lakehouse-uri, depozite de date sau baze de date vectoriale. Proiectul se distinge prin generarea de pipeline-uri bazată pe AI, utilizând modele lingvistice mari pentru a crea codul de extracție și conectorii pentru API-urile REST. De asemenea, suportă stocarea vectorială multimodală și popularea specializată a bazelor de date vectoriale pentru a susține aplicațiile AI și machine learning. Framework-ul acoperă o gamă largă de capabilități, inclusiv evoluția automată a schemei, încărcarea incrementală a datelor prin urmărirea stării și validarea calității datelor prin aplicarea contractelor de date. Oferă instrumente pentru normalizarea datelor relaționale, transformări pre- și post-încărcare și o varietate de adaptoare de destinație pentru baze de date SQL și stocare de obiecte în cloud. Observabilitatea este gestionată prin dashboard-uri de execuție a pipeline-ului, urmărirea lineage-ului coloanelor și verificarea versiunii schemei folosind hash-uri bazate pe conținut.

Botasaurus is a Python web scraping framework and headless browser automation system used to build scalable data extraction tools. It functions as a web data extraction tool and OCR document parser, converting website content, images, and PDF files into structured formats such as JSON, CSV, and Excel. The framework distinguishes itself by providing a scraper management interface that allows Python functions to be wrapped in a web-based UI or deployed as standalone desktop applications. This enables non-technical users to trigger extraction jobs and manage tasks via a graphical interface or RE

This project is an LLM-powered web crawler and data extractor that uses large language models to navigate websites and parse content into structured JSON or Markdown formats. It functions as an automated browser orchestrator and domain discovery engine, interpreting plain English instructions to identify relevant pages and extract specific information. The system distinguishes itself through agentic browser automation, allowing it to perform human-like interactions such as clicking buttons and scrolling based on natural language commands. It employs goal-oriented crawling to analyze website s

Roadmap-Docs is a technical career roadmap repository that provides structured learning paths for software engineering, data science, and artificial intelligence roles. It functions as a professional development curriculum, mapping essential technical milestones and industry-standard tools to guide career advancement. The platform integrates a job market trend analyzer that evaluates industry job postings to identify high-demand skills and competencies. By utilizing an automated data pipeline, the system updates these educational roadmaps to reflect real-time industry requirements and evolvin

Spider is a web-based platform designed for automated data extraction, providing a centralized framework to collect, process, and route structured information from websites. It functions as a comprehensive pipeline that manages the entire lifecycle of data gathering, from initial configuration to final storage in external databases or message queues. The platform distinguishes itself through a visual configuration interface that allows users to define extraction rules and manage scraping templates without writing custom code. It supports both static and dynamic content retrieval by integratin