Awesome GitHub RepositoriesFile-Based Data Sharding

This repository provides the pre-trained neural network and legacy data files used by Tesseract to recognize and extract printed text from images. It serves as a multilingual training data repository and a collection of Long Short-Term Memory models designed for high-accuracy optical character recognition across various global scripts and languages. The data includes specialized models for analyzing image layouts to determine text rotation and script direction. It provides the necessary language-specific datasets and linguistic patterns required to enable Tesseract OCR engines to function. T

This project is a dataset management framework and cross-framework data loader that provides a unified interface for reading data formats compatible with TensorFlow, JAX, and PyTorch. It serves as a library of curated public datasets provided as data streams and includes tools for building, versioning, and documenting large-scale datasets. The system differentiates itself through a distributed data processing engine capable of managing massive datasets across clusters using parallelized pipelines. It utilizes builder-based construction to standardize how data is downloaded and prepared, while

Open Flamingo ist ein multimodales Large-Language-Model-Trainingsframework, das darauf ausgelegt ist, vortrainierte Vision-Encoder mit Sprachmodellen zu integrieren. Es implementiert eine Vision-Language-Architektur, die Cross-Attention-Layer nutzt, um verschachtelte Sequenzen von Bildern und Text zu verarbeiten. Das System zeichnet sich durch seine Few-Shot-multimodalen Lernfähigkeiten aus, die es dem Modell ermöglichen, sich mit einer kleinen Menge an Bild-Text-Beispielen im Prompt an neue visuelle Aufgaben anzupassen. Es unterstützt In-Context-Learning und multimodale Textgenerierung für Aufgaben wie visuelle Fragenbeantwortung (VQA) und Captioning. Das Framework enthält einen verteilten Modell-Trainer, der Datenparallelität und Gradient-Checkpointing zur Speicheroptimierung über mehrere GPUs hinweg einsetzt. Es bietet zudem Utilities für das Laden geshardeter multimodaler Datensätze, parallelisierte Modellevaluierung und Infrastruktur zum Hosten großskaliger Modelle für die Inferenz.

Wikiextractor is a Wikipedia dump parser and dataset preprocessor designed to extract plain text and metadata from MediaWiki database dumps. It functions as a converter that transforms these archives into structured document files or line-delimited JSON objects for use in text corpora and machine learning datasets. The utility includes a MediaWiki template expander that resolves complex template placeholders into their full text representation. It also supports the isolation and extraction of specific individual pages from a full archive without requiring the processing of the entire dataset.