ArchiveBox

ArchiveBox is a self-hosted web archiving system designed to capture and preserve permanent static copies of webpages, media, and PDFs on personal infrastructure. It functions as a digital content curator and personal web archive manager, allowing users to import URLs from bookmarks, RSS feeds, and browser history to create a centralized, searchable knowledge base.

The project is distinguished by its ability to archive private, paywalled, or login-protected content using browser cookies and authenticated session persistence. It ensures long-term availability by saving pages in multiple concurrent formats, including HTML, PDF, and PNG, and can automatically mirror these local snapshots to external preservation services.

The system includes capabilities for multimedia asset extraction, full-text archive indexing, and scheduled content mirroring. Users can manage their collections through a web-based interface, a command-line interface, or a remote API, with options to export the entire collection as a standalone static HTML site for offline browsing.

Features

Digital Preservation Tools - Captures and maintains long-term accessible versions of web content in multiple preservation-grade formats like PDF and PNG.

Web Content Archivers - Captures permanent static copies of webpages, including HTML, JavaScript, PDFs, and media.

Personal Knowledge Management - Collects and organizes bookmarks, RSS feeds, and browser history into a centralized, searchable personal knowledge base.

Content Aggregation & Curation - Collects and organizes content from bookmarks, RSS feeds, and history into a centralized knowledge base.

Session Data Archivers - Uses browser cookies and user profiles to download and store content generated during private web sessions.

Content Formats and Exporting - Saves static copies of web content in HTML, PDF, and PNG formats to ensure long-term availability.

Self-Hosted Web Archivers - Captures and stores permanent static copies of webpages, media, and PDFs on personal infrastructure.

External Feed Integrations - Automatically pulls fresh URLs from RSS feeds on a recurring timetable.

Local Filesystem Storage - Saves archived webpages and media directly to the local server disk using a simple on-disk directory structure.

Web Archive Mirroring - Schedules recurring captures of websites and syncs local archives to external preservation services for redundancy.

Self-Hosted Link Managers - Provides a privately hosted system for managing and preserving collections of web links and static page copies.

Authenticated - Saves pages behind login walls or paywalls using browser cookies and authenticated session persistence.

Paywall Bypass Archiving - Captures pages that require authentication, cookies, or specific login credentials to access.

HTTP Cookie Persistence - Uses saved browser cookies to maintain authenticated sessions and capture private or paywalled web content.

Task Schedulers - Runs archiving tasks on a fixed timetable to automatically capture content.

Web-Based Control Panels - Provides a web-based user interface to control the archiving process and manage collections.

Preservation Formats - Saves webpages in multiple concurrent formats, including HTML, PDF, and PNG, for long-term preservation.

Programmatic Archive Managers - Provides a command-line interface and remote API to control archival tasks and manage tags.

Backup Storage Backends - Provides interfaces to back up preserved archives to remote object storage and external storage backends.

Bulk Media Acquisition - Downloads audio, video, and subtitles, and clones source code repositories for offline storage.

Link Data Imports - Provides utilities for importing web links and URL metadata from various structured formats including CSV and JSON.

Bookmark Importers - Implements importers for collecting web links from browser bookmark files and other external formats.

Third-Party Data Connectors - Utilizes data connectors to ingest web links from third-party archiving services into a local database.

Flat-File Storage - Maintains archives in a simple directory structure on disk to allow browsing without a database.

Full Text Search - Integrates with search backends to enable full-text discovery across the archived content corpus.

Search & Indexing Engines - Integrates external search backends to provide full-text discovery across the local corpus of archived pages.

Cloud Storage - Supports saving archived content to remote cloud storage providers and network shares.

External Archive Mirroring - Automatically sends local snapshots to external preservation services to ensure redundant storage of content.

Automated Asset Extractors - Detects and saves specific assets from pages such as videos, audio, and social media comments.

Archival Collection Managers - Provides a terminal interface for initializing collections and managing archived snapshots.

Environment Variable-Based Configuration - Configures bot behavior and server settings through system environment variables and configuration files.

Asynchronous Task Queues - Processes archiving and import jobs through an asynchronous queue to prevent UI blocking.

Crawler Behavior Configurations - Allows customization of extraction methods and bot identification via configuration files and environment variables.

Extraction Plugins - Employs specialized plugin-based scrapers to detect and download diverse media assets and source code.

Data Management Systems - Self-hosted web archive for creating browsable local backups.

Internet Services - Self-hosted web archiving for creating local browsable backups.

Web Archiving - Platform for archiving and organizing website content.

pirateArchiveBox

Features

Star history