2 个仓库
Awesome GitHub RepositoriesBatch-Driven Data Migration
Manages the migration of massive datasets using controlled batching to optimize memory and avoid packet limits.
Distinct from Large-Scale Dataset Management: Distinct from Large-Scale Dataset Management: focuses on the active process of migration via batching rather than general storage management.
Explore 2 awesome GitHub repositories matching data & databases · Batch-Driven Data Migration. Refine with filters or upvote what's useful.
Awesome Batch-Driven Data Migration GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- zdennis/activerecord-import
zdennis/activerecord-import
4,151在 GitHub 上查看ActiveRecord Import 是一个用于通过单条或批量查询将大型数据集插入数据库的库。它作为一个批量数据导入器和 SQL 批量插入工具，旨在最大限度地减少查询开销并提高写入性能。 该项目包含一个 Upsert 引擎，通过更新现有记录或忽略重复项来处理唯一约束冲突。它还具有递归关联导入器，允许在单次操作中插入父记录及其嵌套的子关联。 该库提供了一个预插入验证层，用于在提交数据前验证记录有效性并报告失败。它通过将数据集拆分为批次来管理大规模数据迁移，以控制内存使用并避免数据库包大小限制。 该工具提供包含成功计数和记录标识符的结果摘要，以及用于数据批次的进度跟踪。
Processes massive datasets in controlled batches to manage memory usage and avoid database packet size limits.
Ruby - cvdfoundation/open-images-dataset
cvdfoundation/open-images-dataset
1,104在 GitHub 上查看This project provides a collection of command-line tools and scripts designed to automate the ingestion, migration, and preparation of large-scale annotated image datasets. It serves as a utility for managing the retrieval of image collections paired with bounding box and segmentation annotations, facilitating their integration into machine learning data pipelines. The toolset enables the bulk transfer of image data and metadata manifests into private cloud storage environments. It utilizes manifest-driven orchestration to process structured resource locations, ensuring that raw image files r
Automates bulk retrieval and migration of large-scale media datasets by processing structured resource manifests.
