Awesome GitHub RepositoriesFile-Based Data Sharding

This repository provides the pre-trained neural network and legacy data files used by Tesseract to recognize and extract printed text from images. It serves as a multilingual training data repository and a collection of Long Short-Term Memory models designed for high-accuracy optical character recognition across various global scripts and languages. The data includes specialized models for analyzing image layouts to determine text rotation and script direction. It provides the necessary language-specific datasets and linguistic patterns required to enable Tesseract OCR engines to function. T

This project is a dataset management framework and cross-framework data loader that provides a unified interface for reading data formats compatible with TensorFlow, JAX, and PyTorch. It serves as a library of curated public datasets provided as data streams and includes tools for building, versioning, and documenting large-scale datasets. The system differentiates itself through a distributed data processing engine capable of managing massive datasets across clusters using parallelized pipelines. It utilizes builder-based construction to standardize how data is downloaded and prepared, while

Open Flamingo es un framework de entrenamiento de modelos de lenguaje multimodal de gran tamaño diseñado para integrar codificadores de visión preentrenados con modelos de lenguaje. Implementa una arquitectura de visión-lenguaje que utiliza capas de atención cruzada para procesar secuencias intercaladas de imágenes y texto. El sistema se caracteriza por sus capacidades de aprendizaje multimodal few-shot, permitiendo al modelo adaptarse a nuevas tareas visuales utilizando un pequeño conjunto de ejemplos de imagen-texto proporcionados en el prompt. Admite aprendizaje en contexto y generación de texto multimodal para tareas como respuesta a preguntas visuales y subtitulado. El framework incluye un entrenador de modelos distribuido que emplea paralelismo de datos y checkpointing de gradiente para la optimización de memoria a través de múltiples GPUs. También proporciona utilidades para la carga de datasets multimodales fragmentados, evaluación de modelos paralelizada e infraestructura para alojar modelos a gran escala para inferencia.

Wikiextractor is a Wikipedia dump parser and dataset preprocessor designed to extract plain text and metadata from MediaWiki database dumps. It functions as a converter that transforms these archives into structured document files or line-delimited JSON objects for use in text corpora and machine learning datasets. The utility includes a MediaWiki template expander that resolves complex template placeholders into their full text representation. It also supports the isolation and extraction of specific individual pages from a full archive without requiring the processing of the entire dataset.