Open-source alternatives to Waybackpack

This project is a specialized TikTok API scraper and data extractor. It functions as a proxy-based web scraper designed to collect user metadata, video posts, and trend feeds, while providing a webhook data pipeline to route scraped information to external URLs via HTTP requests. The tool includes a watermark-free video downloader that saves high-definition content to local storage. It employs cryptographic request signing for server authentication and utilizes session cookie authentication combined with proxy rotation to manage network traffic and avoid rate limits. Capabilities cover bulk

Download complete websites from the Wayback Machine with full asset preservation for offline viewing

OSINT tool to download archived PDF files from archive.org for a given website.

Gau is a command-line tool and passive URL enumerator designed to discover and aggregate known and historical web addresses for specific target domains. It functions as a collection framework that retrieves domain-specific data from public web archives and threat intelligence providers. The tool focuses on passive reconnaissance and open-source intelligence research to map attack surfaces without sending requests directly to target infrastructure. It aggregates data from multiple external sources to identify accessible web endpoints and forgotten pages. The system includes capabilities for r

This tool downloads each page from the Wayback Machine for a specific domain and enables further keyword search on each saved page.

Stream Downloader是一个用于从网络下载媒体资源(视频，音频，图片等)的工具。

Python3 script to download archive.org books in PDF format

SoundCloud (and Bandcamp and Mixcloud) downloader in Python.

API-less slideshare downloader in golang.

This project is a command-line utility designed to fetch video, audio, and image content from a wide range of web platforms. It functions by parsing page metadata and utilizing modular, site-specific scripts to extract direct media stream URLs from complex web structures, enabling the local archiving of digital media for offline use. The tool distinguishes itself through its ability to handle authenticated content, allowing users to inject browser-stored session cookies to access restricted or private media. It also supports real-time media streaming by piping remote content directly into ext

Find way more from the Wayback Machine, Common Crawl, Alien Vault OTX, URLScan, VirusTotal, GhostArchive & Intelligence X!

Weaponizing WaybackUrls for Recon, BugBounties , OSINT, Sensitive Endpoints and what not

A lightweight tool for scraping current and historic Google Analytics data

Spotify-downloader is a command-line utility designed to archive music from Spotify by matching track URLs to external video sources. It functions as a high-fidelity downloader that retrieves audio content and saves it as local files, ensuring optimal sound quality by selecting the highest available bitrate from the source media. The tool distinguishes itself through its ability to maintain local music collections by mirroring remote playlist states. It performs local-remote synchronization to determine which tracks require downloading or removal, while utilizing a modular architecture to dec

gdown is a command-line tool that downloads public files and folders from Google Drive without requiring authentication. It bypasses the mandatory virus-scan warning page to retrieve large files that conventional download tools block, and can resume interrupted transfers using HTTP range requests. Beyond simple file downloads, gdown can recursively download entire folder hierarchies while preserving the local directory structure. It lists the contents of a public folder as structured JSON without downloading the files themselves, and resolves a file's real name and extension without retrievin

Mechanize is a Ruby library for web browser automation and headless browser emulation. It allows for programmatically navigating websites and simulating human behavior without a graphical user interface. The library provides an automated interface for populating and submitting web forms, including text fields, checkboxes, and file uploads. It manages stateful sessions by automatically storing and sending cookies across multiple requests to maintain user authentication and identity. Additional capabilities include web data scraping, the ability to download remote web content, and the maintena

Trafilatura is a Python library and command-line tool for extracting clean, structured text and metadata from web pages. It downloads HTML content, identifies the main body of text, and strips away navigation, ads, and other boilerplate, returning the core article content along with fields like title, author, date, and URL. The tool can also extract user comments and test whether a page contains extractable text, making it a general-purpose web text extraction library. What distinguishes Trafilatura from simpler extractors is its configurable extraction pipeline, which offers high-speed, high