awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
attardi avatar

attardi/wikiextractor

0
View on GitHub↗
3,970 stars·1,006 forks·Python·agpl-3.0·13 views

Wikiextractor

Wikiextractor is a Wikipedia dump parser and dataset preprocessor designed to extract plain text and metadata from MediaWiki database dumps. It functions as a converter that transforms these archives into structured document files or line-delimited JSON objects for use in text corpora and machine learning datasets.

The utility includes a MediaWiki template expander that resolves complex template placeholders into their full text representation. It also supports the isolation and extraction of specific individual pages from a full archive without requiring the processing of the entire dataset.

The system handles large-scale data processing through stream-based XML parsing and regex-based markup stripping to produce clean text. Extracted data is organized via document sharding and exported as JSON containing article IDs, revision IDs, URLs, titles, and body text.

Features

  • Wikipedia Tools - Converts Wikipedia database dumps into plain text files containing individual documents for each article.
  • Corpus Preprocessing - Cleans MediaWiki markup from raw dumps to create a text-only corpus for training machine learning models.
  • Stream-Based Parsing - Processes large database dumps sequentially using an event-driven parser to avoid loading entire files into memory.
  • Wikipedia Content Expansion - Replaces MediaWiki template placeholders with full content to ensure accurate text representation of articles.
  • Plain Text Extraction - Parses database backup files to remove markup and save cleaned plain text into manageable files.
  • Wiki Template Resolvers - Resolves complex MediaWiki templates into their full text representation during the extraction process.
  • Database Dump Parsers - Extracts plain text and metadata from MediaWiki database dumps into structured document files.
  • Markup Stripping - Uses regular expression patterns to strip wiki-specific formatting and metadata to produce clean plain text.
  • Recursive Template Resolution - Resolves complex MediaWiki template placeholders by recursively fetching and inserting the corresponding content.
  • Page Content Retrievals - Enables the retrieval and cleaning of individual Wikipedia articles from a full archive.
  • Direct-Access Data Extraction - Implements a mechanism to isolate specific articles from a dump file by skipping to the relevant byte offset.
  • Single-Record Extraction - Isolates and processes specific pages from a larger dump file instead of the entire dataset.

Star history

Star history chart for attardi/wikiextractorStar history chart for attardi/wikiextractor

How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Open-source alternatives to Wikiextractor

Similar open-source projects, ranked by how many features they share with Wikiextractor.
  • searxng/searxng-dockersearxng avatar

    searxng/searxng-docker

    3,157View on GitHub↗

    This project is a containerized search infrastructure designed to deploy a privacy-focused metasearch engine. It acts as a self-hosted search proxy that aggregates results from multiple external web, image, and academic search providers while anonymizing requests and stripping trackers to protect user identity. The system utilizes Docker to orchestrate the search instance, integrating caching mechanisms and reverse proxy support to ensure a private and efficient search environment. It employs a modular adapter-based integration to standardize diverse external API responses and a processing pi

    dockerdocker-compose
    View on GitHub↗3,157
  • wikimedia/wikipedia-ioswikimedia avatar

    wikimedia/wikipedia-ios

    3,341View on GitHub↗

    The Wikipedia iOS app is a native client for reading, editing, and contributing to Wikipedia articles on iPhone and iPad. It provides a bridge-based rendering system that serializes view updates over JSON to native iOS components for fast scrolling, along with a native wikitext editor featuring real-time syntax highlighting and formatting tools for editing article content. The app includes a feature flag system with compile-time and runtime flags for enabling experimental capabilities through a hidden settings panel, and a file-based localization system that loads language-specific resource fi

    Swiftobjective-cswiftwikipedia
    View on GitHub↗3,341
  • box/spoutbox avatar

    box/spout

    4,229View on GitHub↗

    Spout is a spreadsheet file processing library and multi-format generator designed for reading and writing CSV, XLSX, and ODS files. It functions as a stream-based parser that processes large spreadsheet files incrementally to avoid loading entire documents into memory. The library provides capabilities for programmatic spreadsheet generation and data extraction. It supports custom content styling, allowing for the application of fonts, backgrounds, borders, and number formats to individual cells or rows. Beyond basic file input and output, the project covers workbook manipulation through se

    PHPcsvfastmemory-efficient
    View on GitHub↗4,229
  • apache/tikaapache avatar

    apache/tika

    3,572View on GitHub↗

    Tika is a content analysis toolkit and Java library designed for detecting and extracting metadata and text from thousands of different file types. It functions as a universal document text extractor and metadata extraction engine, converting complex files into plain text or XHTML. The system employs a specialized MIME type detector that identifies document formats using magic bytes and metadata to determine the correct parser. It serves as an OCR integration gateway, connecting to external text recognition tools to extract content from image files. The project covers a broad range of extrac

    Javacontentextractionjava
    View on GitHub↗3,572
See all 30 alternatives to Wikiextractor→

Frequently asked questions

What does attardi/wikiextractor do?

Wikiextractor is a Wikipedia dump parser and dataset preprocessor designed to extract plain text and metadata from MediaWiki database dumps. It functions as a converter that transforms these archives into structured document files or line-delimited JSON objects for use in text corpora and machine learning datasets.

What are the main features of attardi/wikiextractor?

The main features of attardi/wikiextractor are: Wikipedia Tools, Corpus Preprocessing, Stream-Based Parsing, Wikipedia Content Expansion, Plain Text Extraction, Wiki Template Resolvers, Database Dump Parsers, Markup Stripping.

What are some open-source alternatives to attardi/wikiextractor?

Open-source alternatives to attardi/wikiextractor include: searxng/searxng-docker — This project is a containerized search infrastructure designed to deploy a privacy-focused metasearch engine. It acts… wikimedia/wikipedia-ios — The Wikipedia iOS app is a native client for reading, editing, and contributing to Wikipedia articles on iPhone and… box/spout — Spout is a spreadsheet file processing library and multi-format generator designed for reading and writing CSV, XLSX,… apache/tika — Tika is a content analysis toolkit and Java library designed for detecting and extracting metadata and text from… joshclose/csvhelper — CsvHelper is a library for reading and writing comma-separated value files by mapping data to custom class objects. It… alibaba/easyexcel — EasyExcel is a Java processing library designed for reading and writing XLS, XLSX, and CSV files. It functions as a…