Awesome GitHub RepositoriesData Loading Extraction

Datasets is a library designed for the management, processing, and sharing of large-scale data collections for machine learning workflows. It functions as both a data processing framework and a versioning platform, providing tools to organize, filter, and transform massive datasets while ensuring reproducibility across research and development teams. The library distinguishes itself by enabling the handling of datasets that exceed available system memory. It utilizes memory-mapped file access, disk-based caching, and lazy iterative streaming to maintain performance when working with large-sca

Faker is a Python library designed to generate realistic synthetic data for software testing, database prototyping, and privacy-preserving anonymization. It provides a comprehensive suite of tools to create diverse information types, including personal identities, financial records, geographic locations, and technical system metadata, allowing developers to populate environments with mock data that mimics real-world structures. The library is built on a modular provider architecture that supports dynamic method dispatch, enabling users to extend functionality by registering custom data genera

csvkit is a composable Unix-style command-line toolkit for converting, filtering, and analyzing CSV files directly from the terminal. It provides a suite of focused single-purpose commands that can be combined via pipes to build complex data processing workflows, with a modular architecture that includes a column-type inference engine for automatically detecting data types and a streaming-pipeline design for efficient handling of tabular data. The toolkit distinguishes itself through its SQL-engine abstraction layer, which allows users to run SQL queries directly against CSV files without req

Snorkel is a weak supervision system that enables users to programmatically generate training labels for machine learning models without manual annotation. At its core, it provides a framework for writing labeling functions as Python callables that each vote on data points, and then trains a probabilistic graphical model over these multiple weak supervision sources to estimate latent true labels without any ground truth data. The system automatically learns accuracy and correlation parameters between labeling functions by analyzing observed agreement patterns on unlabeled data, converting lab

xmltodict es una librería de Python que proporciona serialización bidireccional entre documentos XML y diccionarios. Funciona como un analizador (parser) que convierte entradas marcadas en pares clave-valor y como una utilidad de serialización que transforma diccionarios de nuevo en documentos XML estructurados. El proyecto incluye un procesador de flujo incremental que utiliza callbacks basados en profundidad para manejar archivos XML grandes manteniendo un uso de memoria constante. Cuenta con un gestor de espacios de nombres (namespaces) para asignar prefijos y declaraciones, así como un sanitizador de seguridad que bloquea la expansión de entidades externas y valida nombres de elementos para prevenir ataques de inyección. La librería proporciona capacidades para la aplicación de tipos de datos, como forzar que elementos específicos se representen como listas independientemente del número de hijos. También admite el post-procesamiento de datos a través de callbacks definidos por el usuario y ofrece controles configurables para expandir, colapsar u omitir espacios de nombres durante el proceso de conversión.

gdown is a command-line tool that downloads public files and folders from Google Drive without requiring authentication. It bypasses the mandatory virus-scan warning page to retrieve large files that conventional download tools block, and can resume interrupted transfers using HTTP range requests. Beyond simple file downloads, gdown can recursively download entire folder hierarchies while preserving the local directory structure. It lists the contents of a public folder as structured JSON without downloading the files themselves, and resolves a file's real name and extension without retrievin

Tablib es una biblioteca de Python diseñada para importar, exportar y manipular conjuntos de datos tabulares. Funciona como un convertidor y gestor de datos multiformato, permitiendo a los usuarios mover información entre diferentes estándares de archivos. La biblioteca admite la transformación de datos a través de formatos CSV, JSON, YAML y Excel. Proporciona una interfaz programática para gestionar estos conjuntos de datos añadiendo filas, filtrando columnas y segregando registros. El sistema utiliza una representación interna común y un mapeo basado en adaptadores para normalizar diversas fuentes de entrada. Esto permite rutinas de lectura y escritura consistentes en todos los formatos de archivo admitidos.

Textract is a multi-format text extraction tool and parser. It provides a unified interface to extract plain text from a variety of sources, including documents, images, and audio files. The system functions as a document content parser for PDFs and spreadsheets, an image text extractor using optical character recognition, and a speech-to-text transcriber for audio recordings.

img2dataset es un pipeline de datasets de imágenes de alto rendimiento y herramienta de preprocesamiento diseñada para descargar y procesar millones de imágenes desde URLs para el entrenamiento de machine learning. Funciona como un descargador de imágenes distribuido y exportador de datos a almacenamiento en la nube, moviendo grandes datasets visuales desde fuentes web directamente a formatos estructurados. El sistema prioriza la adquisición de datos de alto rendimiento distribuyendo cargas de trabajo entre múltiples núcleos de CPU y máquinas. Se integra directamente con buckets de almacenamiento en la nube remotos y emplea un sistema de seguimiento basado en manifiestos para reanudar descargas interrumpidas sin reprocesar los datos existentes. La herramienta proporciona una suite de preprocesamiento completa para la preparación de datasets de machine learning, incluyendo redimensionamiento de imágenes, recorte y filtrado de propiedades basado en tamaño o relación de aspecto. También verifica la integridad de la imagen mediante comparación de hash y asegura el cumplimiento de las directivas de robots durante el flujo de trabajo de scraping. El proyecto está implementado en Python.

Camelot is a Python library and processing engine designed to extract tabular data from PDF documents. It converts unstructured tables into machine-readable formats such as CSV, JSON, and Excel. The project provides specialized toolsets for different document types, using line detection for ruled tables and whitespace analysis for borderless tables. It includes an optical character recognition system to recover structured data from image-based scanned PDFs that lack a digital text layer. The library handles complex document layouts, including encrypted files, rotated pages, and tables that s

Synthetic data generation for tabular data

xlwings - Make Excel fly with Python!

Extract data from a wide range of Internet sources into a pandas DataFrame.

python-magic is a C-binding wrapper that provides a Python interface for the libmagic system library. It functions as a file signature analyzer and MIME type detector, identifying file formats by comparing header bytes against a database of known binary signatures. The library enables the identification of file types from both file paths and raw data buffers. It supports custom file signature matching through the injection of user-provided magic databases, allowing for the detection of specialized or proprietary formats. The project covers binary data analysis and MIME type mapping to transl

Please use openpyxl where you can...

Singer is an open source standard for moving data between databases, web APIs, files, queues, and just about anything else you can think of. The Singer spec describes how data extraction scripts — called “Taps” — and data loading scripts — called “Targets” — should communicate using a standard…

Intake is a lightweight package for finding, investigating, loading and disseminating data.

Convert CSV files into a SQLite database. Browse and publish that SQLite database with Datasette.

Data search & enrichment library for Machine Learning → Easily find and add relevant features to your ML & AI pipeline from hundreds of public and premium external data sources, including open & commercial LLMs