dimitri/pgloader

Features

• PostgreSQL Bulk Importers - pgloader loads CSV, fixed-width, DBF, and IXF files into PostgreSQL with configurable parsing, transformation, and error handling.
• PostgreSQL Target Migrations - Transfers schema and data from various sources into PostgreSQL automatically.
• Bulk Data Loading - Migrates data from various database and file formats into PostgreSQL using the COPY command.
• Cross-Database Data Migrations - Migrates entire databases between engines with a single unattended command.
• Cross-Database Schema Migrations - Transfers complete database schemas between different database systems during migration.
• Parallel Batch Loading Pipelines - pgloader reads source data in concurrent reader-writer pipelines, each processing a disjoint range, and sends formatted batches to PostgreSQL via COPY.
• Migration Casting Rules - Applies custom type casting, field renaming, and data reformatting rules during migration.
• Database Connectivity - Connects to source and target databases using connection strings with authentication and options.
• MS SQL to PostgreSQL Migrations - Migrates MS SQL Server databases to PostgreSQL with schema discovery.
• SQLite to PostgreSQL Migrations - Migrates entire SQLite databases into PostgreSQL with schema and data transfer.
• SQLite to PostgreSQL Migrations - Automates migration of SQLite databases into PostgreSQL with schema discovery and index creation.
• Database Schema Migrations - Transfers complete schema definitions including indexes, foreign keys, and comments.
• Schema Introspection - Connects to live source databases to introspect metadata, then generates target schema DDL and transformation rules programmatically.
• PostgreSQL-to-PostgreSQL Migrations - Migrates complete PostgreSQL databases between servers with all objects.
• Data Loading - Reads data from CSV, fixed-width, dBase, and IBM IXF files, transforming it on the fly.
• Error-Resilient Loading - Continues loading valid rows while saving rejected records to separate files, using per-row error handling instead of full transaction rollbacks.
• Multi-Format File Loading - Loads data from CSV, fixed-column, dBase, and IBM IXF files using the streaming COPY protocol.
• Streaming COPY Protocol Implementations - Streams transformed data rows directly into PostgreSQL's COPY protocol, bypassing row-by-row INSERT for high-throughput bulk loading.
• MySQL Compatibility - Loads MySQL databases into PostgreSQL with schema and parallel data loading.
• MySQL to PostgreSQL Migrations - Migrates MySQL databases to PostgreSQL with type casting and sanitization.
• Cross-Database Schema Replicators - Discovers and replicates source database schemas including tables, indexes, and constraints.
• SQLite to PostgreSQL Migrations - Migrates entire SQLite databases into PostgreSQL with schema creation.
• CSV Data Loaders - Reads structured data from CSV files and inserts it into PostgreSQL tables using the COPY command.
• Data Transformation Engines - Applies custom type casting, field renaming, and data reformatting rules during migration.
• Database Migration CLIs - Command-line tool that automates schema discovery and parallel loading from multiple source databases.
• Schema Migrations - Controls table creation, index building, and sequence resetting during automated schema migration.
• Streaming Data Transformers - Modifies and cleans data values as they are read from a source file before inserting them into PostgreSQL.
• File-to-PostgreSQL Migrations - Loads data from multiple file formats into PostgreSQL using streaming COPY.
• Database Load Pipelines - Runs concurrent reader-writer pipelines per table to maximize PostgreSQL bulk load throughput.
• Parallel Data Pipelines - Spawns multiple concurrent threads that each read a disjoint range of source data, transform it, and feed it into a dedicated COPY stream.
• Drop and Create Migrations - Drops existing target objects before recreating them, enabling safe repeated migration runs.
• Pre and Post Load SQL - Executes custom SQL commands before and after loading data, enabling table creation and post-processing.
• Continuous Synchronization - Keeps a target PostgreSQL database synchronized with a source database by streaming changes over time.
• Configurable CSV Parsing - Specifies field delimiters, enclosing characters, escape sequences, and header handling to match CSV file formats.
• Data Reformatters - Transforms incompatible data formats, such as zero-dates, during the loading process.
• Load-Time Transformations - Applies custom rewriting rules to source data, such as converting integers to IP ranges or zero dates to NULL, before loading.
• Custom Casting Rules - Allows defining custom rules for converting source data types to PostgreSQL types during migration.
• Field Mapping Expressions - Transforms and projects input fields into a different set of target columns using custom expressions.
• CSV - Applies per-column options such as date format parsing, null-value substitution, and whitespace trimming during CSV loading.
• Fixed-Width File Loaders - Loads data from fixed-width text files into PostgreSQL tables.
• Inline and Stdin - Reads COPY-formatted data directly from within the command file or from standard input.
• PostgreSQL COPY Format Loading - Loads data from PostgreSQL COPY TEXT format files, supporting local files, HTTP URLs, and zip archives.
• Stream and Inline Sources - Reads data from a file, stdin, or inline source and loads it into a PostgreSQL table.
• Migration DSLs - pgloader uses a dedicated command language to specify every aspect of a data load, including casting rules, encoding overrides, and pre/post SQL.
• Online Schema Alterations During Migration - pgloader renames, drops, or adds columns and tables on the fly to fix past design issues during migration.
• Pre and Post Load SQL Execution - Executes custom SQL queries before or after the data load to prepare or finalize the database.
• Data-Only Migrations - Loads data into an existing schema with index management for performance.
• Drop-Create Schema Migrations - Drops and recreates target tables, indexes, and constraints before loading data for repeatable migrations.
• Pre and Post Load SQL - Executes user-defined SQL commands before and after a data load for table creation, indexing, or post-processing.
• Schema or Data Only Migrations - Loads only schema or only data with index management for performance.
• Index Creation Concurrency Limiters - pgloader caps the number of parallel CREATE INDEX threads to avoid overwhelming the database during bulk loads.
• Schema - Renames source schemas and tables on the fly, mapping them to different names in the target database.
• Schema Renaming - Renames tables, moves them to different schemas, or sets storage parameters before migration.
• Multi-Source CSV Loading - Loads data from a file path, standard input, inline data, or files matching a regex pattern.
• Drop-Create Migrations - Drops and recreates target schema objects on each run for safe repeated migrations during development and testing.
• On-the-Fly Renaming - Renames tables, columns, or constraints on the fly as part of the migration process.
• Pre and Post Load SQL Execution - Executes user-defined SQL statements before and after the load to prepare tables or create indexes.
• Legacy Database File Loaders - Loads data from legacy dBase file format into PostgreSQL tables.
• Citus Cluster Migrations - Migrates data into Citus distributed PostgreSQL clusters with automatic shard distribution.
• Bulk Load Error Recovery - Continues loading valid rows while saving rejected records to separate files for review.
• Selective Table Migrations - Selectively migrates a subset of tables from source to target database.
• Intra-Table Parallel Pipelines - pgloader runs multiple independent reader-writer pipelines per table when the source supports range partitioning and concurrency exceeds one.
• Table-Level Concurrency Controls - pgloader sets how many tables copy simultaneously, with separate defaults for database and file sources.
• Error Handling - Continues loading valid records while saving rejected rows to separate reject files.
• Bulk Load - Offers configurable error handling during bulk loading, continuing or stopping on errors.
• Data Utilities - High-speed data loading utility.
• Databases - High-performance data loading tool for PostgreSQL.
• Databases and ORMs - High-performance data loading tool for PostgreSQL.

Star history