Awesome GitHub RepositoriesData Loading Extraction

Datasets is a library designed for the management, processing, and sharing of large-scale data collections for machine learning workflows. It functions as both a data processing framework and a versioning platform, providing tools to organize, filter, and transform massive datasets while ensuring reproducibility across research and development teams. The library distinguishes itself by enabling the handling of datasets that exceed available system memory. It utilizes memory-mapped file access, disk-based caching, and lazy iterative streaming to maintain performance when working with large-sca

Faker is a Python library designed to generate realistic synthetic data for software testing, database prototyping, and privacy-preserving anonymization. It provides a comprehensive suite of tools to create diverse information types, including personal identities, financial records, geographic locations, and technical system metadata, allowing developers to populate environments with mock data that mimics real-world structures. The library is built on a modular provider architecture that supports dynamic method dispatch, enabling users to extend functionality by registering custom data genera

csvkit is a composable Unix-style command-line toolkit for converting, filtering, and analyzing CSV files directly from the terminal. It provides a suite of focused single-purpose commands that can be combined via pipes to build complex data processing workflows, with a modular architecture that includes a column-type inference engine for automatically detecting data types and a streaming-pipeline design for efficient handling of tabular data. The toolkit distinguishes itself through its SQL-engine abstraction layer, which allows users to run SQL queries directly against CSV files without req

Snorkel is a weak supervision system that enables users to programmatically generate training labels for machine learning models without manual annotation. At its core, it provides a framework for writing labeling functions as Python callables that each vote on data points, and then trains a probabilistic graphical model over these multiple weak supervision sources to estimate latent true labels without any ground truth data. The system automatically learns accuracy and correlation parameters between labeling functions by analyzing observed agreement patterns on unlabeled data, converting lab

xmltodict est une bibliothèque Python qui permet une sérialisation bidirectionnelle entre des documents XML et des dictionnaires. Elle fonctionne comme un analyseur qui convertit une entrée balisée en paires clé-valeur et comme un utilitaire de sérialisation qui transforme les dictionnaires en documents XML structurés. Le projet inclut un processeur de flux incrémental qui utilise des rappels (callbacks) basés sur la profondeur pour traiter de gros fichiers XML tout en maintenant une utilisation constante de la mémoire. Il dispose d'un gestionnaire d'espaces de noms pour mapper les préfixes et les déclarations, ainsi qu'un désinfecteur de sécurité qui bloque l'expansion d'entités externes et valide les noms d'éléments pour prévenir les attaques par injection. La bibliothèque offre des capacités de forçage de types de données, comme imposer que certains éléments soient représentés sous forme de listes quel que soit le nombre d'enfants. Elle prend également en charge le post-traitement des données via des rappels définis par l'utilisateur et offre des contrôles configurables pour développer, réduire ou ignorer les espaces de noms pendant le processus de conversion.

gdown is a command-line tool that downloads public files and folders from Google Drive without requiring authentication. It bypasses the mandatory virus-scan warning page to retrieve large files that conventional download tools block, and can resume interrupted transfers using HTTP range requests. Beyond simple file downloads, gdown can recursively download entire folder hierarchies while preserving the local directory structure. It lists the contents of a public folder as structured JSON without downloading the files themselves, and resolves a file's real name and extension without retrievin

Tablib est une bibliothèque Python conçue pour importer, exporter et manipuler des jeux de données tabulaires. Elle fonctionne comme un convertisseur et gestionnaire de données multi-formats, permettant aux utilisateurs de déplacer des informations entre différents standards de fichiers. La bibliothèque prend en charge la transformation de données à travers les formats CSV, JSON, YAML et Excel. Elle fournit une interface programmatique pour gérer ces jeux de données en ajoutant des lignes, en filtrant des colonnes et en séparant des enregistrements. Le système utilise une représentation interne commune et un mapping basé sur des adaptateurs pour normaliser diverses sources d'entrée. Cela permet des routines de lecture et d'écriture cohérentes à travers les formats de fichiers pris en charge.

Textract est un outil d'extraction et un parseur de texte multi-format. Il fournit une interface unifiée pour extraire du texte brut à partir d'une variété de sources, y compris des documents, des images et des fichiers audio. Le système fonctionne comme un parseur de contenu de documents pour les PDF et les feuilles de calcul, un extracteur de texte d'image utilisant la reconnaissance optique de caractères (OCR), et un transcripteur parole-texte pour les enregistrements audio.

img2dataset is a high-performance image dataset pipeline and preprocessing tool designed to download and process millions of images from URLs for machine learning training. It functions as a distributed image downloader and cloud storage data exporter, moving large visual datasets from web sources directly into structured formats. The system prioritizes high-throughput data acquisition by distributing workloads across multiple CPU cores and machines. It integrates directly with remote cloud storage buckets and employs a manifest-based tracking system to resume interrupted downloads without re

Camelot is a Python library and processing engine designed to extract tabular data from PDF documents. It converts unstructured tables into machine-readable formats such as CSV, JSON, and Excel. The project provides specialized toolsets for different document types, using line detection for ruled tables and whitespace analysis for borderless tables. It includes an optical character recognition system to recover structured data from image-based scanned PDFs that lack a digital text layer. The library handles complex document layouts, including encrypted files, rotated pages, and tables that s

Synthetic data generation for tabular data

xlwings - Make Excel fly with Python!

Extract data from a wide range of Internet sources into a pandas DataFrame.

python-magic is a C-binding wrapper that provides a Python interface for the libmagic system library. It functions as a file signature analyzer and MIME type detector, identifying file formats by comparing header bytes against a database of known binary signatures. The library enables the identification of file types from both file paths and raw data buffers. It supports custom file signature matching through the injection of user-provided magic databases, allowing for the detection of specialized or proprietary formats. The project covers binary data analysis and MIME type mapping to transl

Please use openpyxl where you can...

Singer is an open source standard for moving data between databases, web APIs, files, queues, and just about anything else you can think of. The Singer spec describes how data extraction scripts — called “Taps” — and data loading scripts — called “Targets” — should communicate using a standard…

Intake is a lightweight package for finding, investigating, loading and disseminating data.

Convert CSV files into a SQLite database. Browse and publish that SQLite database with Datasette.

Data search & enrichment library for Machine Learning → Easily find and add relevant features to your ML & AI pipeline from hundreds of public and premium external data sources, including open & commercial LLMs