apache/hive

Features

• Hadoop - Provides a SQL-on-Hadoop data warehouse for querying petabytes of distributed data.
• Distributed SQL Querying - Executes SQL queries against petabytes of data in distributed storage for batch analytics.
• Hadoop Data Warehouses - Provides a SQL-on-Hadoop data warehouse that queries and manages petabytes of data stored in distributed storage.
• Thrift-Based Query Executions - Provides a Thrift-based service for executing SQL queries from remote clients with authentication.
• Concurrent Read-Write Transactions - Executes concurrent read and write transactions on tables with optimistic or pessimistic locking and lockless reads.
• Data Lake Transaction Managers - Implements an ACID transaction manager for concurrent reads and writes on ORC tables.
• ORC Transaction Managers - Provides ACID transactions on ORC tables for concurrent read-write operations.
• Columnar Storage Engines - Uses the Optimized Row Columnar (ORC) format to improve read, write, and compression performance.
• Metastore Integrations - Provides a central metadata repository serving table schemas to Hive, Spark, and Impala.
• Hive Metastore Services - Provides a central metadata repository that stores table schemas and partition information for Hive and other data lake tools.
• Cloud Native Object Storage - Natively supports S3, Azure Data Lake, and Google Cloud Storage for cloud-native data warehousing.
• Automatic Partition Insertions - Automatically creates partitions during data insertion, eliminating manual partition management.
• Partition Key Filters - Restricts query scans to specific partitions using predicates on partition key columns.
• Data Serialization Formats - Converts data between storage and in-memory representations using SerDes for multiple formats.
• SQL Data Loading and Transformation - Loads data into tables and transforms it using INSERT, UPDATE, DELETE, and scripting operations.
• SQL Data Analysis - Provides a SQL interface for ad-hoc querying, summarization, and analysis of large datasets.
• Table Schemas - Creates, renames, adds columns to, and drops tables and partitions while preserving schema.
• Tez-Based Query Executions - Executes SQL queries using the Tez framework for lower latency than MapReduce.
• Database Object Management - Creates, alters, drops, and describes databases, tables, views, functions, indexes, and partitions.
• Data Lake Table Metadata Stores - Stores and serves table and partition metadata via a service API for Hive, Spark, Impala, and other data lake tools.
• SQL Query Interfaces - Queries files stored in HDFS, HBase, or other storage systems through a unified SQL interface.
• Federated Data Query Engines - Links Hive to remote databases via data connectors for federated query execution across disparate sources.
• File-Based Data Import - Imports data from HDFS files into Hive tables using external table definitions or load commands.
• Key-Based Partitioning - Divides table data into storage units based on column values for faster query analysis.
• Low-Latency Analytical Queries - Delivers sub-second query responses by reusing persistent query infrastructure and optimized data caches.
• In-Memory Caching Engines - Provides an in-memory caching engine for sub-second query responses on large datasets.
• Parquet Data Processing - Reads and writes data in the Parquet columnar format for efficient storage and querying.
• Multi-Engine Execution Backends - Executes queries on Spark, Tez, or MapReduce to balance performance and resource usage.
• Cost-Based Optimizers - Uses a cost-based optimizer with table statistics and materialized views for query planning.
• Low-Latency Query Engines - Ships the LLAP interactive engine for sub-second query responses on large datasets.
• Remote Table Connectivity - Maps remote databases or schemas to Hive databases, making all tables queryable without manual mapping.
• Row Aggregations - Combines multiple input rows into a single output row, computing statistics like count, sum, average, and percentile.
• Data Lake - Provides a central metastore for managing schemas across Hive, Spark, Impala, and other tools.
• SQL Query Optimizations - Uses a cost-based optimizer to automatically improve SQL query performance and resource usage.
• Managed and External Tables - Defines tables whose data is either managed by Hive or stored at a user-specified location.
• Query Caching Daemons - Runs persistent daemon workers that cache data in memory for sub-second SQL query responses.
• SQL Query Executions - Provides a REST endpoint for executing Hive queries and returning results.
• WebHCat - Ships a WebHCat REST API for submitting Hive, MapReduce, and Pig jobs and managing HCatalog metadata.
• Apache Thrift Integrations - Uses Apache Thrift for remote procedure calls to submit queries to a Hive instance.
• Thrift RPC Servers - Implements a Thrift RPC server that accepts SQL queries and returns results over binary or HTTP transport.
• Thrift Service Implementations - Implements a Thrift-based server that accepts SQL queries from remote clients with authentication.
• Directed Acyclic Graph Execution Engines - Uses a DAG execution engine (Tez) for lower-latency interactive SQL queries on Hadoop.
• Authorization Mode Managers - Restricts user actions on data and metadata through configurable authorization modes.
• Enterprise Security Integrations - Integrates with Kerberos, LDAP, SAML, JWT, OAuth2, and Apache Ranger for enterprise-grade security.
• Multi-Method API Authentication - Supports SAML, JWT, Kerberos, and LDAP authentication for HiveServer2 and the metastore.
• Enterprise Access Controls - Integrates Kerberos, Apache Ranger, and Apache Atlas for enterprise-grade access control.
• Schema-on-Read Formats - Applies a schema to various file formats like CSV, Parquet, and ORC, enabling SQL queries on unstructured data.
• Table Bucketing - Distributes data across fixed buckets using hash of a column for efficient sampling and query pruning.
• Window Ranking Functions - Applies analytic functions like RANK, ROW_NUMBER, and SUM over a defined window of rows.
• BI Tool Integration - Provides JDBC and ODBC drivers that connect BI tools directly to Hive for querying.
• RCFile Data Access - Stores and retrieves table data in the Record Columnar File format for Hive tables.
• Druid Datasource Querying - Indexes data from Hive into Druid and runs SQL queries against Druid datasources directly.
• Database Namespace Organization - Groups tables, views, and partitions into namespaces to avoid naming conflicts and enforce security boundaries.
• Interactive - Executes HiveQL statements from a shell utility in both interactive and batch script modes.
• Iceberg Table Lifecycle Management - Manages large analytic datasets in Iceberg format with snapshot isolation, branching, tagging, and full DML support.
• Common Table Expressions - Defines a temporary result set within a WITH clause that can be referenced by a single SELECT, INSERT, or CREATE statement.
• ACID Table Compaction Optimizations - Rebalances and prioritizes compaction requests for ACID and Iceberg tables to improve storage and query performance.
• Continuous Data Imports - Streams small batches of records into an existing Hive partition or table, making them immediately visible to subsequent queries.
• Custom MapReduce Script Embedding - Embeds user-provided mapper and reducer scripts within SQL queries using the TRANSFORM clause.
• Storage Compaction Utilities - Rebalances and prioritizes compaction of ACID and Iceberg tables to optimize storage and query performance.
• Catalog Integration - Connects to an Iceberg REST catalog secured with OAuth2 to manage tables with snapshots and branches.
• Data Import and Export - Exports and imports table or partition data and metadata between Hive instances.
• Spark Execution Backends - Supports running Hive queries on Apache Spark for accelerated performance.
• S3 Querying - Declares tables over S3 data and runs SQL queries against it using EC2 compute resources.
• Cross-Cluster Table Replication - Bootstraps and checkpoints replication of tables, including external tables, with tracking metrics for consistency.
• Avro Table Reads and Writes - Reads and writes Avro-encoded data as Hive tables, inferring the table schema from the Avro schema and supporting nested structures.
• Beeline Query Execution - Connects to HiveServer2 via the Beeline CLI client to run SQL queries interactively.
• Map Join Optimizations - Automatically selects map joins for star-schema queries by loading smaller tables into memory as hash tables.
• Predicate Pushdown Joins - Applies filter predicates to all joined tables to avoid scanning unnecessary partitions.
• Date and Time Libraries - Provides built-in SQL functions for parsing, formatting, and extracting date and time components.
• Disaster Recovery Solutions - Performs bootstrap and incremental replication for robust backup and business continuity.
• SQL Geospatial Analysis - Processes spatial data using ESRI-based functions for geometry operations and analysis.
• Multi-Dimensional Aggregations - Computes subtotals and grand totals across multiple dimensions using CUBE, ROLLUP, and GROUPING SETS operators.
• Tez Execution Backends - Supports running queries on Apache Tez for lower-latency DAG-based execution.
• Materialized Views - Pre-computes and stores summary results, then rewrites incoming queries to use those summaries for faster execution.
• Pluggable Connector Frameworks - Provides a pluggable connector framework for mapping external systems like HBase and Druid to Hive tables.
• Distributed Row Distributions - Orders or distributes rows across reducers using ORDER BY, SORT BY, DISTRIBUTE BY, or CLUSTER BY clauses.
• Custom SQL Functions - Adds user-defined functions, aggregates, and table functions to SQL for custom data processing.
• Custom Aggregation Function Writing - Extends built-in aggregation by implementing user-defined aggregation functions in Java.
• Custom Scalar Function Writing - Extends built-in SQL functions by implementing user-defined scalar functions in Java.
• Procedural SQL Execution - Executes procedural SQL code, including PL/SQL and T-SQL, on Hive and other engines.
• Cost-Based Compilation Optimizations - Compiles SQL queries using materialized views, anti-joins, branch pruning, and histogram statistics for faster execution.
• Pluggable Serialization Frameworks - Provides a pluggable SerDe layer that supports multiple storage formats like ORC, Parquet, and Avro.
• Procedural SQL Execution - Executes procedural SQL code alongside standard queries for complex data transformations.
• Query Result Persistence - Writes the output of a SELECT statement into another table, a local file, or a distributed filesystem directory.
• Union All Result Combinations - Merges the result sets of multiple SELECT statements into a single result set using UNION.
• Transactional Data Mutations - Inserts, updates, and deletes individual records in ACID tables without rewriting entire partitions.
• Iceberg Table Ingestion - Supports Apache Iceberg tables natively via a storage handler for cloud-native, high-performance data lake architectures.
• JDBC External Table Connectors - Creates external tables that read from any JDBC data source with predicate pushdown support.
• Kudu Table Connectors - Provides a storage handler for querying Apache Kudu tables directly from Hive.
• Table Function Invocation - Applies a user-defined table-generating function to each input row and joins the output rows to the original input.
• SerDe-Based Format Plugins - Uses a custom SerDe to serialize and deserialize table data in any format for reading and writing.
• Metadata REST Endpoints - Provides a REST API for managing table metadata and submitting Hadoop jobs.
• Beeline Shell Connectivity - Provides a JDBC-based command-line shell for connecting to HiveServer2 in embedded or remote mode.
• Compaction Worker Pools - Routes compaction requests to dedicated worker pools so high-priority tables are processed without waiting in the default queue.
• JDBC Driver Loading - Ships a JDBC driver that enables external applications to connect and run queries against Hive.
• Conditional Logic - Provides built-in SQL functions for conditional logic like CASE, IF, and NULL handling.
• Automatic Query-Time Decompression - Imports text files compressed with Gzip or Bzip2 and decompresses them automatically during query execution.
• SQL Type Conversions - Ships a comprehensive set of SQL type conversion functions for query-time data transformation.
• SQL Mathematical Functions - Provides a wide range of mathematical functions callable directly within SQL queries.
• Sampling Retrievals - Returns a random sample of rows from a table by selecting a specific bucket.
• String Manipulation - Provides built-in SQL functions for string concatenation, extraction, encoding, and pattern matching.
• HBase Table Connectors - Accesses HBase tables with HiveQL for both reading and writing data alongside native tables.
• Multi-Table Insertions - Sends the results of a single query to several tables or HDFS files in one pass.
• Big Data Storage - Data warehouse software for querying and managing large distributed datasets.
• Data Engineering - Data warehouse software for managing large datasets.

Star history