pudo/dataset
Dataset
Este proyecto es una capa de acceso a datos SQL y generador de esquemas que permite leer y escribir registros en bases de datos relacionales tratando las tablas como estructuras de datos simples. Funciona como un generador de esquemas automático que crea tablas y columnas de base de datos sobre la marcha según la estructura de los datos entrantes.
La herramienta proporciona un cargador masivo de alto rendimiento que importa grandes conjuntos de datos utilizando transacciones atómicas agrupadas para asegurar la consistencia de los datos. También incluye un mecanismo de upsert de registros que determina si actualizar una fila existente o insertar una nueva según identificadores únicos.
El sistema cubre la gestión dinámica de esquemas, incluyendo la resolución implícita de columnas y el aprovisionamiento de tablas. Además, proporciona una interfaz de consulta basada en colecciones para recuperar registros o extraer valores únicos sin escribir consultas manuales.
Features
- Dynamic Column Resolutions - Detects new fields in incoming data and modifies the database schema to accommodate them without manual migrations.
- SQL Data Access Layers - Read and write records by treating tables as simple data structures instead of writing manual queries.
- Dynamic Schema Evolution - Automatically creating and updating database tables and columns on the fly based on the structure of incoming data.
- SQL Translation Layers - Provides a translation layer that allows records to be read and written by treating database tables as simple data structures.
- Schema-on-Write Generators - Create tables and columns automatically when writing data to a destination that does not yet exist.
- Bulk Data Loading - Efficiently importing large sets of records into a database using bulk loading and transaction support.
- Collection Querying - Wraps SQL SELECT statements in simplified methods to retrieve full tables or unique column values.
- Data Abstraction Layers - Maps database rows to simple data structures to allow record manipulation without writing manual SQL queries.
- Upsert Operations - Implements a mechanism to update existing rows or insert new ones based on the presence of unique identifiers.
- On-the-Fly Table Creations - Automatically creates database tables and columns based on the structure of the data being inserted at runtime.
- Dynamic Column Provisioning - Automatically generates and modifies database tables and columns on the fly based on the structure of the incoming data.
- Dynamic Column Resolution - A tool that creates SQL tables and columns on the fly based on the structure of the data being written.
- Upsert Operations - Update existing rows or insert new ones based on whether a matching record exists in the table.
- Bulk Loaders - Groups multiple data insertions into a single atomic transaction to increase write throughput and ensure data consistency.
- Transactional Updates - Groups multiple data insertions into a single transaction to increase write performance and ensure atomic updates.
- Database Transaction Wrappers - Wraps multiple database operations into atomic transactions to ensure data consistency and increase write performance during bulk loading.
- Upsert Operations - Checks for existing records using unique identifiers to determine whether to update a row or insert a new one.
- Database Clients - Data handling for SQL stores.
- Database ORMs - Listed in the “Database ORMs” section of the Awesome Python awesome list.
- Database Tools - JSON-like interface for working with SQL databases.
- Object Relational Mappers - Stores Python dicts in relational databases.
- ORM Frameworks - Stores dictionaries in databases.
