perspective-dev/perspective
Perspective
Perspective is a columnar data analytics engine and high-performance visualization component powered by WebAssembly. It provides a system for analyzing and visualizing large or streaming datasets through interactive data grids and charts, utilizing a compiled binary to achieve near-native performance within the browser.
The project distinguishes itself through a WebSocket-based data streaming interface and deep Apache Arrow integration, which minimize memory overhead when synchronizing tables between servers and clients. It acts as a remote query proxy capable of translating visualization configurations into native queries for external databases such as ClickHouse and DuckDB.
The engine covers a broad range of analysis capabilities, including group-by aggregations, dataset joining, pivoting, and row filtering. It supports complex data management through incremental updates, primary key handling, and reactive view propagation that automatically updates visualizations as source data changes.
The system can be deployed as a browser-only execution environment, a server-replicated setup, or as embeddable interactive widgets within notebook environments like JupyterLab.
Features
- High-Performance Visualizers - Implements a high-performance WebAssembly-powered UI component for rendering interactive data grids and charts for large-scale datasets.
- WASM-Powered Visualization Components - Provides a high-performance UI component for analyzing and visualizing large or streaming datasets using a WebAssembly-powered engine.
- Remote Connectivity - Establishes a WebSocket connection to a remote server to perform queries and analysis on remote data.
- In-Memory Streaming Analytics - Creates in-memory columnar tables that ingest and query large-scale streaming datasets.
- Apache Arrow Processing - Uses the high-performance Apache Arrow columnar memory format to transfer large datasets between servers and clients.
- Binary Serialization - Uses a columnar memory format to efficiently transfer and serialize data between different runtime environments.
- In-Memory Columnar Accelerators - Stores strongly typed data in columns to enable high-performance aggregation and filtering of large datasets in memory.
- Columnar Data Processors - Implements a query engine that performs aggregations, filtering, and pivoting on strongly typed columnar data.
- Columnar Storage Engines - Implements a storage engine optimized for analytical workloads by organizing data in a columnar format.
- Interactive Data Grids - Ships a high-performance grid component for tabular data exploration with real-time updates and sorting.
- Runtime-to-Runtime Synchronization - Duplicates and synchronizes data tables between server and browser clients using the Apache Arrow format.
- Reactive View Bindings - Automatically recalculates aggregates and filters in dependent visualizations when the underlying source data changes.
- Group-By Aggregations - Computes high-performance aggregations on large or streaming datasets to summarize data by specific dimensions.
- Incremental Data Appending - Supports adding new records to existing tables to grow datasets over time via incremental updates.
- Incremental Data Streaming - Processes incremental data streams to efficiently recalculate aggregates and update visualizations in real time.
- Interactive Visualization Rendering - Renders dynamic charts and grids that update in response to user input through a framework-agnostic interface.
- Real-Time Plot Updates - Builds interactive dashboards that update instantly as streaming data flows into a WebAssembly-powered system.
- Real-time Data Synchronization - Reflects data source changes in real time by incrementally recalculating aggregates and filters in the view.
- WebAssembly Engines - Implements a high-performance query engine compiled to WebAssembly for near-native data analysis directly in the browser.
- Remote Query Execution - Connects a browser frontend to a server-side engine via WebSockets to analyze data stored on a remote server.
- Dataset Analysis - Enables immediate grouping, sorting, filtering, and aggregations on datasets via an interactive viewer.
- External Query Translation - Translates high-level visualization configurations into native queries for external data engines without loading full datasets into memory.
- Virtual Query Abstractions - Translates internal query operations into native calls for external databases without loading full datasets into memory.
- Transient In-Memory Tables - Constructs transient in-memory tables from input data to serve as the foundation for real-time analysis.
- Multi-Chart Type Libraries - Supports a wide array of chart types including line, bar, scatter, heatmap, treemap, and candlestick.
- Analytics Server Hosting - Establishes a server-side connection via WebSockets to receive and process data requests from a front-end viewer.
- Binary Data Streaming - Synchronizes server-side tables and query results to browser clients via a real-time binary message bus.
- WebSocket Servers - Implements a server-side WebSocket connection to process binary messages from front-end viewers without timeouts.
- Cross-Language Bridges - Transfers data between JavaScript and WebAssembly runtimes using a binary columnar format for zero-copy efficiency.
- Large Dataset Explorers - Performs complex queries and aggregations on massive datasets in the browser without crashing the user interface.
- Real-Time Data Streaming - Synchronizes server-side tables and query results to browser clients via a real-time binary message bus.
- WebSocket Dashboards - Synchronizes server-side data tables with browser viewers using a real-time binary message bus over WebSockets.
- Notebook Analytics - Provides specialized widgets within Jupyter notebooks to explore and visualize dataframes and Arrow data interactively.
- Computed Columns - Defines new data columns using custom expressions for complex calculations and boolean filtering logic.
- Remote DataFrame Querying - Exposes in-memory Polars DataFrames to browser clients over a WebSocket connection for remote analysis.
- Data Schema Definitions - Provides declarative systems for defining columnar data structures and types to hold analysis data.
- Cross-Client Synchronization - Mirrors server-side data tables across multiple concurrent browser clients to keep all viewers synchronized in real time.
- Data Visualization Dashboards - Provides tools for organizing multiple visualization components into analytical dashboards.
- ClickHouse Connectors - Provides specialized connectors for establishing high-performance connections to ClickHouse analytical database servers.
- Conditional Row Filters - Restricts dataset output to rows meeting specific criteria across multiple columns using boolean logical operators.
- DataFrame Ingestion - Converts pandas or polars DataFrame objects into internal high-performance tables while preserving indexing.
- Dataset Joins - Combines multiple data sources using inner, left, or right joins based on shared keys.
- Dataset Pivoting - Splits a dataset into multiple columns based on unique field values without aggregating data.
- Primary Keys - Designates specific columns as unique identifiers to enable in-place updates and targeted row deletions.
- Virtual Database Integration - Integrates external DuckDB instances as virtual servers to query and visualize data natively.
- Python Table Hosting - Hosts tables on a Python server to make them accessible to JavaScript clients over a WebSocket connection.
- Reactive Joins - Combines two source tables on a shared key to create a reactive read-only joined table.
- Multi-Format Data Loading - Loads data from multiple formats including CSV, JSON, and Apache Arrow into high-performance internal tables.
- Parallel Execution - Executes analytics operations across multiple threads using an internal pool to accelerate data processing.
- Logic Thread Isolation - Offloads data processing to background worker threads to ensure the user interface remains responsive during heavy analysis.
- Execution Environment Configurations - Provides flexible configuration between browser-only execution, server replication, or server-side query execution.
- Reactive Change Detection - Implements reactive change detection that triggers callbacks automatically when table data is modified or deleted.
- Notebook Analysis Widgets - Provides interactive widgets for transforming and visualizing large datasets within JupyterLab environments.
- Embedded Analytics Interfaces - Integrates professional data grids and charting components directly into web applications for end users.
- Analysis Widgets - Embeds an interactive analysis widget into notebook environments for rapid exploration of large datasets.
- Synchronized View Filtering - Implements mechanisms to coordinate filtering state across multiple independent visualization panels.
- Virtualized Data Rendering - Streams only the data required for the current view to efficiently render massive datasets without crashing the browser.
- Browser Database Support - Executes complex analytics queries against WASM-powered databases directly within the client's browser.
- Web Worker Orchestrators - Offloads heavy calculations and data accumulation to background Web Workers to keep the user interface responsive.
- Data Visualization - High-performance streaming data visualization and analytics component.
