Awesome GitHub RepositoriesMetadata Extraction

Readability is a JavaScript library designed for web content extraction. It functions as a DOM parsing utility and article metadata extractor that isolates the primary text of a webpage by removing clutter such as advertisements and navigation bars. The library employs a heuristic-based content detector to predict if a webpage contains a parseable article before performing full extraction. It uses a parsing workflow to convert complex HTML documents into a simplified format, facilitating the implementation of distraction-free reader views. The tool covers several capability areas, including

Acest proiect este un instrument de verificare a e-mailurilor care verifică dacă o adresă de e-mail există și poate primi mesaje fără a trimite un e-mail real. Oferă aceste capabilități printr-un API HTTP programatic și o interfață CLI locală. Sistemul se distinge prin combinarea verificării handshake-ului SMTP și a rezoluției înregistrărilor DNS cu un instrument de analiză a riscurilor care detectează adresele temporare, conturile bazate pe roluri și domeniile catch-all. Include, de asemenea, un agregator de metadate pentru a prelua informații publice de profil și imagini asociate cu o anumită adresă de e-mail. Suprafața de capabilități mai largă acoperă validarea livrabilității e-mailurilor, evaluarea riscurilor pentru a preveni înregistrările frauduloase și testarea conectivității SMTP. Serviciul backend poate fi implementat ca un serviciu în rețea folosind un model de deployment containerizat.

Kreuzberg is a document extraction engine that converts PDFs, Office files, images, and over 90 other formats into clean, structured text and metadata. It is built around a compiled Rust core that can be used as a native library, a command-line tool, a REST API server, or a WebAssembly module for browser-based processing. The system is designed to run entirely on self-hosted infrastructure, with no data leaving the user's environment. What distinguishes Kreuzberg is its breadth of integration surfaces and its pipeline architecture. It exposes extraction capabilities through native bindings fo

This project is a comprehensive resource directory for web data extraction, providing a curated collection of tools and libraries for parsing data, automating browsers, and managing network operations. It serves as a guide for extracting structured information from HTML, XML, JSON, and PDF formats. The toolkit focuses on advanced data collection strategies, including headless browser automation to interact with JavaScript and a suite of network utilities for DNS resolution and WebSocket connections. It specifically covers methods for bypassing bot protections through proxy pool management, us

Blackbird is an open-source OSINT investigation tool that searches across hundreds of online platforms to discover accounts linked to a given username or email address. It functions as a username and email search engine, consolidating discovered profiles into a single list with low false positives for investigative analysis. The tool incorporates an AI-enhanced profile analyzer that uses a built-in AI API to generate behavioral and technical summaries of discovered online profiles. It also provides a documentation query interface that accepts natural-language questions via HTTP GET requests t

Trafilatura is a Python library and command-line tool for extracting clean, structured text and metadata from web pages. It downloads HTML content, identifies the main body of text, and strips away navigation, ads, and other boilerplate, returning the core article content along with fields like title, author, date, and URL. The tool can also extract user comments and test whether a page contains extractable text, making it a general-purpose web text extraction library. What distinguishes Trafilatura from simpler extractors is its configurable extraction pipeline, which offers high-speed, high

Acest proiect este un scraper specializat pentru API-ul TikTok și un extractor de date. Funcționează ca un web scraper bazat pe proxy, conceput pentru a colecta metadate ale utilizatorilor, postări video și feed-uri de trenduri, oferind în același timp un pipeline de date prin webhook pentru a direcționa informațiile extrase către URL-uri externe prin cereri HTTP. Instrumentul include un downloader de videoclipuri fără watermark care salvează conținutul la înaltă definiție în stocarea locală. Utilizează semnarea criptografică a cererilor pentru autentificarea serverului și folosește autentificarea prin cookie-uri de sesiune combinată cu rotația de proxy-uri pentru a gestiona traficul de rețea și a evita limitările de rată (rate limits). Capabilitățile acoperă extracția de conținut în masă prin pipeline-uri de procesare batch și recuperarea metadatelor pentru profiluri, hashtag-uri și piese muzicale. Datele extrase sunt exportate în formate structurate JSON sau CSV, iar urmărirea stării locale este utilizată pentru a înregistra istoricul descărcărilor și a preveni cererile duplicate.

This project is a multi-purpose REST API utility collection and developer suite. It serves as a centralized service for real-time information aggregation, data transformation, and a wide array of programmatic tools. The service distinguishes itself by providing a broad range of specialized content delivery endpoints, from curated daily summaries and global trending rankings to randomized entertainment content like jokes and quotes. It also functions as a real-time aggregator for environmental and network data, including weather forecasts, currency exchange rates, and public IP lookups. The c

This project is a desktop-based bibliographic reference manager designed to organize academic research libraries and automate citation workflows. It functions as a research assistant that integrates directly with word processors and text editors, enabling users to insert and format references while writing. The application is built on a Java-based portable runtime, allowing it to operate as a self-contained tool that stores preferences and data in local configuration files. The platform distinguishes itself through a modular plugin architecture and a commitment to human-readable, text-based f

Acest proiect este un scraper web neautentificat, conceput pentru a extrage date publice din API-ul frontend Twitter. Acesta funcționează ca un extractor de date din rețelele sociale care simulează cereri de browser pentru a colecta informații fără a fi nevoie de chei API oficiale sau autentificarea contului de utilizator. Instrumentul oferă capabilități pentru colectarea postărilor publice, recoltarea metadatelor profilurilor de utilizator (cum ar fi biografii și locații) și preluarea subiectelor în tendințe (trending topics) clasificate pe regiuni geografice. Poate efectua scraping de conținut vizat pe baza unor nume de utilizator, hashtag-uri sau interogări de căutare specifice. Sistemul gestionează colectarea datelor prin paginare, emularea clientului bazată pe headere pentru a imita browserele web și parsarea răspunsurilor JSON structurate din endpoint-uri publice.

HaE is a network traffic analysis tool designed to extract, classify, and highlight specific data fragments within network messages and HTTP traffic. It functions as an HTTP data extractor and traffic content filter, utilizing a network metadata aggregator to centralize highlighted data fragments and annotations for analysis. The tool identifies high-value network packets by mapping classification results to visual color markers and employs a modular classification system to isolate data fragments from binary or text streams. It distinguishes the severity of matched data by piping extracted c

This project is a Go library and command-line utility designed for the retrieval and local archival of remote video content. It provides a programmatic interface for fetching media streams, allowing users to extract metadata and download video files directly to local storage. The library distinguishes itself through its ability to resolve playback restrictions by performing algorithmic transformations on obfuscated authentication tokens. This signature decryption process enables the tool to bypass standard access limitations, while its interface-driven design allows for the selection of speci

GeneralNewsExtractor is a specialized system for identifying and extracting structured news data through configurable HTML filtering and XPath-based parsing. It functions as an automated news scraper and web page content parser that isolates article bodies and metadata from news webpages. The system uses YAML or JSON configuration files to manage extraction rules and XPath definitions, decoupling the processing logic from site-specific selectors. This allows for targeted noise removal and the isolation of titles, authors, and publication dates through customizable filtering. The tool covers

FOCA is a digital forensics metadata analyzer and open-source intelligence tool used to extract hidden information from various document types. It functions as a metadata extraction tool that isolates technical data and EXIF information from PDFs, office documents, and SVG files. The system integrates an open-source intelligence scanner that identifies and downloads target files from the web using multiple search engine APIs. This allows for the automated discovery and acquisition of remote web assets for batch analysis and digital evidence gathering. The software provides capabilities for d

probe-rs is an embedded debugging toolset used for flashing, debugging, and inspecting microcontrollers via hardware debug probes. It functions as an embedded firmware flasher, a memory manager for reading and writing to processor registers, and a server implementing the Debug Adapter Protocol to connect hardware debugging to various code editors. The project includes a CMSIS-Pack target parser to extract device metadata and flash configurations from industry-standard definition files. It provides a driver-based probe abstraction to interact with various hardware debug probes regardless of th

CeWL is a custom wordlist generator and web crawling security tool designed to extract unique words and metadata from websites. It functions as an OSINT metadata extractor and security scanner, identifying potential passwords and usernames by analyzing HTML and JavaScript content. The tool differentiates itself by combining recursive spidering with metadata extraction, allowing it to collect email addresses, author names, and creator metadata from web pages and linked files. It also captures domains, subdomains, and path components to include in generated lists. Broad capabilities include we