Awesome GitHub RepositoriesData Loading Extraction

Datasets is a library designed for the management, processing, and sharing of large-scale data collections for machine learning workflows. It functions as both a data processing framework and a versioning platform, providing tools to organize, filter, and transform massive datasets while ensuring reproducibility across research and development teams. The library distinguishes itself by enabling the handling of datasets that exceed available system memory. It utilizes memory-mapped file access, disk-based caching, and lazy iterative streaming to maintain performance when working with large-sca

Faker is a Python library designed to generate realistic synthetic data for software testing, database prototyping, and privacy-preserving anonymization. It provides a comprehensive suite of tools to create diverse information types, including personal identities, financial records, geographic locations, and technical system metadata, allowing developers to populate environments with mock data that mimics real-world structures. The library is built on a modular provider architecture that supports dynamic method dispatch, enabling users to extend functionality by registering custom data genera

csvkit is a composable Unix-style command-line toolkit for converting, filtering, and analyzing CSV files directly from the terminal. It provides a suite of focused single-purpose commands that can be combined via pipes to build complex data processing workflows, with a modular architecture that includes a column-type inference engine for automatically detecting data types and a streaming-pipeline design for efficient handling of tabular data. The toolkit distinguishes itself through its SQL-engine abstraction layer, which allows users to run SQL queries directly against CSV files without req

Snorkel is a weak supervision system that enables users to programmatically generate training labels for machine learning models without manual annotation. At its core, it provides a framework for writing labeling functions as Python callables that each vote on data points, and then trains a probabilistic graphical model over these multiple weak supervision sources to estimate latent true labels without any ground truth data. The system automatically learns accuracy and correlation parameters between labeling functions by analyzing observed agreement patterns on unlabeled data, converting lab

xmltodict 是一个 Python 库，提供 XML 文档与字典之间的双向序列化功能。它既是一个将标记化输入转换为键值对的解析器，也是一个将字典转换回结构化 XML 文档的序列化工具。 该项目包含一个增量流处理器，使用基于深度的回调来处理大型 XML 文件，同时保持恒定的内存占用。它具有用于映射前缀和声明的命名空间管理器，以及一个安全清理器，可阻止外部实体扩展并验证元素名称以防止注入攻击。 该库提供了数据类型强制功能，例如强制将特定元素表示为列表（无论子元素数量如何）。它还支持通过用户定义的回调进行数据后处理，并为转换过程中的命名空间扩展、折叠或跳过提供了可配置的控制选项。

gdown is a command-line tool that downloads public files and folders from Google Drive without requiring authentication. It bypasses the mandatory virus-scan warning page to retrieve large files that conventional download tools block, and can resume interrupted transfers using HTTP range requests. Beyond simple file downloads, gdown can recursively download entire folder hierarchies while preserving the local directory structure. It lists the contents of a public folder as structured JSON without downloading the files themselves, and resolves a file's real name and extension without retrievin

Tablib 是一个用于导入、导出和操作表格数据集的 Python 库。它作为多格式数据转换器和管理器，允许用户在不同的文件标准之间迁移信息。 该库支持 CSV、JSON、YAML 和 Excel 格式之间的数据转换。它提供了一个编程接口，通过添加行、过滤列和分离记录来管理这些数据集。 系统使用通用的内部表示和基于适配器的映射来规范化不同的输入源，从而在支持的文件格式之间实现一致的读写操作。

Textract 是一个多格式文本提取工具和解析器。它提供了一个统一的接口，用于从各种来源（包括文档、图像和音频文件）中提取纯文本。 该系统作为 PDF 和电子表格的文档内容解析器、使用光学字符识别 (OCR) 的图像文本提取器，以及音频录音的语音转文本转录器。

img2dataset 是一个高性能图像数据集流水线和预处理工具，旨在为机器学习训练从 URL 下载并处理数百万张图像。它作为一个分布式图像下载器和云存储数据导出器，将大型视觉数据集从 Web 源直接移动到结构化格式中。 该系统通过在多个 CPU 核心和机器之间分配工作负载，优先考虑高吞吐量的数据获取。它直接与远程云存储桶集成，并采用基于清单的追踪系统，无需重新处理现有数据即可恢复中断的下载。 该工具为机器学习数据集准备提供了完整的预处理套件，包括图像缩放、裁剪以及基于尺寸或长宽比的属性过滤。它还通过哈希比较验证图像完整性，并在抓取工作流期间确保符合机器人协议。 该项目使用 Python 实现。

Camelot is a Python library and processing engine designed to extract tabular data from PDF documents. It converts unstructured tables into machine-readable formats such as CSV, JSON, and Excel. The project provides specialized toolsets for different document types, using line detection for ruled tables and whitespace analysis for borderless tables. It includes an optical character recognition system to recover structured data from image-based scanned PDFs that lack a digital text layer. The library handles complex document layouts, including encrypted files, rotated pages, and tables that s

Synthetic data generation for tabular data

xlwings - Make Excel fly with Python!

Extract data from a wide range of Internet sources into a pandas DataFrame.

python-magic is a C-binding wrapper that provides a Python interface for the libmagic system library. It functions as a file signature analyzer and MIME type detector, identifying file formats by comparing header bytes against a database of known binary signatures. The library enables the identification of file types from both file paths and raw data buffers. It supports custom file signature matching through the injection of user-provided magic databases, allowing for the detection of specialized or proprietary formats. The project covers binary data analysis and MIME type mapping to transl

Please use openpyxl where you can...

Singer is an open source standard for moving data between databases, web APIs, files, queues, and just about anything else you can think of. The Singer spec describes how data extraction scripts — called “Taps” — and data loading scripts — called “Targets” — should communicate using a standard…

Intake is a lightweight package for finding, investigating, loading and disseminating data.

Convert CSV files into a SQLite database. Browse and publish that SQLite database with Datasette.

Data search & enrichment library for Machine Learning → Easily find and add relevant features to your ML & AI pipeline from hundreds of public and premium external data sources, including open & commercial LLMs