awslabs/aws-data-wrangler
Aws Data Wrangler
This project is an AWS pandas integration library and data pipeline framework designed to simplify the movement and transformation of data between local memory and AWS storage and analytics services. It functions as a cloud data lake toolkit and storage file manager, allowing users to read, write, and transform structured data across various cloud environments.
The library distinguishes itself as a distributed compute orchestrator capable of managing clusters in environments such as EMR to process datasets that exceed the memory limits of a single machine. It also provides specialized capabilities for managing vector indices and performing similarity searches within cloud storage buckets.
Its broader capability surface covers cloud database ETL for services like DynamoDB, RDS, and Timestream, as well as cloud data catalog management via AWS Glue. It supports serverless data analytics through Athena and Redshift, and provides utilities for managing S3 objects, indexing documents in OpenSearch, and analyzing CloudWatch logs.
Features
- Cloud Data Lake Integrations - Simplifies reading and writing structured data frames to S3 using formats like Parquet, CSV, and JSON.
- Pandas - Bridges cloud-native data lake storage with pandas dataframes for efficient data loading and representation.
- DataFrame Cloud Integrations - Simplifies the movement and transformation of data between local memory data frames and AWS cloud services.
- AWS Data Retrieval - Provides a unified interface for extracting large datasets from Athena and Redshift into pandas data frames.
- AWS Pandas Integration Libraries - Extends pandas functionality to provide direct integration with AWS storage and analytics services.
- Cloud Data Ingestion Interfaces - Provides a unified interface for transferring pandas data frames into various AWS managed services.
- Cloud Database ETL - Enables efficient movement of structured data between pandas and AWS databases including DynamoDB, RDS, and Timestream.
- Analytical Data Loads from Object Storage - Facilitates loading data from cloud object storage into analytical engines for extraction and transformation workflows.
- Compute Cluster Orchestration - Controls the lifecycle and configuration of remote compute clusters to process datasets exceeding local memory.
- Data Catalogs - Implements programmatic management of AWS Glue databases, tables, and partitions to ensure data discoverability.
- Data Lake Orchestrators - Provides high-performance movement and orchestration of data between cloud-native data lakes and warehouses.
- Optimized DataFrame Storage - Saves structured data as optimized files in cloud buckets and optionally registers them in a data catalog.
- Data Warehouse Integrations - Synchronizes data frames with warehouse clusters using direct SQL queries or staged transfers.
- Cloud Data Warehouse Connectivity - Provides secure connectivity to load data warehouse query results into data frames.
- Database Catalog Management - Enables the management of structural metadata, including tables and partitions, within a centralized data catalog.
- Distributed Data Processing - Distributes data processing tasks across multiple compute nodes to handle datasets exceeding single-node memory.
- NoSQL Data Frame Integrations - Enables reading, writing, and deleting items in NoSQL document tables using data frames.
- Relational Database Integrations - Integrates pandas data frames with relational databases for reading and writing via SQL.
- Serverless Query Integrations - Executes SQL queries against serverless query engines and returns the result sets as data frames.
- SQL Query Execution Engines - Processes standard SQL commands against serverless engines to load results into data frames.
- Cloud Object Storage Clients - Implements client-side libraries for uploading, downloading, and managing files and directories within cloud storage buckets.
- Cloud File Writing - Saves data frames to cloud storage buckets in multiple common file formats.
- Toolkits - Offers a comprehensive set of tools for managing S3 files, Glue catalogs, and Athena queries using data frames.
- Data Lake Table Management - Creates and repairs metadata tables while managing insertions and deletions for large-scale data lake tables.
- Vector Index Mappers - Maps structured data frames to high-dimensional vector indices for similarity searches within cloud storage.
- Python Data Pipeline Frameworks - Provides a Python-based framework for orchestrating data movement between memory and cloud warehouses.
- Search and Indexing - Indexes structured files and data frames into search engines for efficient retrieval via queries.
- Temporary File Staging - Moves data between memory and warehouses by utilizing intermediate files in cloud storage for bulk loads.
- Time-Series Data Modeling - Supports structuring and querying time-stamped data frames within time-series databases.
- Vector Indices - Manages high-dimensional vector indices in cloud storage for performing similarity searches.
- Metadata Synchronizers - Automatically synchronizes table metadata with central catalogs when writing data frames to cloud storage.
- Data API Integrations - Executes SQL queries and inserts data into database instances using a managed cloud data API.
- Remote Ray Cluster Connections - Connects to and initializes remote Ray clusters to distribute heavy data processing tasks.
- Cloud Query Engine Normalizers - Transforms column names and data types into formats required by specific cloud query engines and catalogs.
- Data Ingestion Pipelines - Utility belt for handling data operations on AWS.
- MLOps and Deployment - Pandas-like interface for AWS data services.
- Cloud Infrastructure - Simplifies data integration and ETL tasks on AWS.
- Data Science Tools - Pandas extension for AWS data services.
Star history
How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
