rerun-io/rerun
Rerun
Rerun is a multimodal data visualizer and robotics data logger designed for rendering synchronized streams of 3D spatial data, images, and time-series metrics. It functions as a tool for capturing high-frequency sensor data and AI outputs into a queryable columnar format, providing a dedicated interface for viewing MCAP recording files and analyzing physical environments.
The project distinguishes itself as a machine learning dataset streamer, capable of feeding logged recordings directly into GPU buffers and PyTorch training pipelines without intermediate exports. It supports a high-performance data pipeline that includes on-the-fly decompression and random seeking to streamline the transition from data logging to model training.
The platform covers broad capability areas including 3D spatial scene rendering, geospatial mapping, and the visualization of images and tensors. It provides tools for temporal data management and timeline synchronization, alongside SQL-based querying for extracting specific data segments from large-scale recordings.
The visualization interface can be hosted as a standalone viewer or embedded directly into native application windows and notebooks.
Features
- Training Data Pipelines - Streams logged recordings directly into PyTorch or GPU buffers to eliminate manual data export steps.
- 3D Scene Renderers - Provides a high-performance 3D renderer for point clouds, meshes, and camera poses to analyze physical environments.
- Robotics Visualization Tools - Visualizes synchronized sensor streams, 3D point clouds, and coordinate transforms to analyze physical robotic agents.
- Dataset Loaders - Feeds logged recordings directly into PyTorch and GPU training pipelines as efficient dataset loaders.
- Multimodal AI Systems - Captures and stores synchronized images, point clouds, and time series for debugging embodied AI systems.
- Spatial Pose Management - Defines coordinate frames and spatial poses to position entities accurately in 3D space.
- Stream Synchronization Utilities - Assigns deterministic timestamps across concurrent data streams to ensure synchronized playback of multimodal sensor data.
- Robotics and Embodied AI - Provides tools for logging and reviewing multimodal sensor data to identify failures in embodied AI models.
- File Format Processing - Parses channels, schemas, and statistics from MCAP recording files for analysis.
- Columnar Storage Engines - Uses a column-oriented storage format for multimodal time-series data to enable efficient querying and random access.
- Temporal Stream Synchronization - Renders synchronized streams of images and point clouds to analyze multimodal sensor data over a shared timeline.
- Time-Series Visualizers - Provides interactive scalar line series and bar charts to track numerical metrics over time.
- Multimodal Sequence Visualizers - Renders multimodal data in sync to scrub through episodes and compare sensors over time.
- ML Dataset Lazy-Loading - Streams recordings directly into machine learning frameworks via on-the-fly decompression and random seeking.
- Local Data Persistence - Saves high-frequency sensor logs and AI outputs to binary files for offline playback and sharing.
- MCAP File Integration - Loads timestamped messages from MCAP containers via file import or SDK integration for visualization.
- MCAP File Viewers - Provides a dedicated visualization interface for importing and parsing timestamped messages from MCAP recording files.
- Message Decoders - Decodes raw MCAP messages using reflection or archetypes to transform them into queryable components.
- Real-Time Data Streaming - Feeds real-time sensor data from webcams and depth sensors directly into the viewer for instant monitoring.
- Robotics Data Loggers - Captures and persists high-frequency sensor data and AI outputs into a queryable columnar format.
- Temporal Data Management - Assigns timestamps and durations to logged data to ensure deterministic playback and animation.
- 2D Spatial Rendering - Renders 2D spatial primitives including points, lines, boxes, and arrows within a coordinate system.
- Robotics-Specific - Renders domain-specific robotics data like occupancy grids and state changes on a synchronized timeline.
- Image Data Visualizers - Visualizes multi-dimensional tensors and various image types, including depth and segmentation maps.
- Multimodal Visualizers - Renders synchronized streams of 3D spatial data, images, and time-series metrics for AI and robotics analysis.
- Video Streaming - Manages and displays raw video streams and individual frame references synchronized with other sensor data.
- Geospatial Visualizations - Visualizes geographic data using latitude and longitude for mapping robot trajectories and locations.
- Sensor Data Visualizers - Visualizes high-frequency sensor telemetry and AI outputs to monitor the state of physical agents.
- Real-Time Monitoring Systems - Streams live telemetry and high-frequency sensor data to a remote viewer for real-time system inspection.
- Timeline Synchronization - Assigns time values to logged data across different threads to ensure temporal alignment during playback.
- Coordinate Transform Hierarchies - Manages spatial entities by logging a tree of relative poses to position 3D objects in a global coordinate system.
- Data Loaders - Integrates logged datasets into ML workflows using dedicated loaders to feed data into models.
- Dataset Integration - Exposes recording files as iterable datasets with random seeks and on-the-fly decompression for PyTorch frameworks.
- Multimodal Dataset Catalogs - Provides a searchable catalog and cloud storage for organizing large-scale multimodal recordings for team collaboration.
- Data Integration And Import - Allows importing and visualizing recorded data from external formats like images, video, and point clouds via plugins.
- Data Ingestion - Imports large datasets using record batches to move structured columnar data efficiently into the system.
- GPU-Accelerated Data Streams - Feeds column-aware and video-codec-aware data from recordings directly to GPUs to eliminate intermediate export steps.
- Data Query Management - Includes tools for retrieving and transforming multimodal data from a catalog for local analysis and team management.
- Data Sinking - Provides mechanisms for directing logged sensor data to various external destinations including gRPC servers and binary files.
- High-Volume Data Ingestion - Provides high-volume data ingestion capabilities to record multimodal sensor data from multiple threads and processes simultaneously.
- gRPC Data Streaming - Transmits logged data from producers to remote viewers in real-time using a high-performance gRPC framework.
- Robotic Dataset Catalogs - Organizes robotic data using a scalable catalog with version control and cloud storage for team access.
- Schema Mapping Engines - Decodes raw binary messages into queryable components using predefined data models and reflection for robotics types.
- Search and Indexing - Provides searchable indexes to locate specific byte ranges within large-scale recordings for efficient retrieval.
- Byte-Range Indexing - Catalogs large-scale datasets in object storage using index offsets to fetch specific data segments without full downloads.
- SQL Query Interfaces - Provides a SQL interface to query recorded data directly without requiring manual data exports.
- Structured Data Extraction - Extracts information from recorded data streams into structured dataframes for detailed analysis.
- Time-Series SQL Querying - Executes SQL queries over recording files to retrieve specific data columns and time ranges.
- Object Storage Indexing - Catalogs petabyte-scale multimodal data in object storage using byte-range reads for efficient querying.
- Spatial Data Analysis Tools - Renders 2D and 3D geometric primitives and geospatial coordinates on a shared temporal timeline.
- Remote Monitoring Transmission - Sends logged sensor data to remote viewers in real-time using high-performance network protocols.
- Remote Access Connectivity - Establishes network connections to remote viewer instances for real-time data streaming and visualization.
- Application Embedding Interfaces - Hosts the visualization interface directly within a host application process by embedding the renderer into a native window.
- Robotic Data Processors - Loads and transforms robotic sensor data streams using a declarative language to normalize nested structures.
- Annotation Collaboration - Allows teams to explore and annotate datasets within a shared viewing environment to identify and trace failures.
- Dataset Sharing - Provides a unified viewer for teams to collectively explore and annotate shared recording datasets.
- Catalog Management - Organizes multimodal logs into queryable segments using a dedicated catalog server.
- Embedded Interfaces - Integrates the visualization interface into notebooks or web pages for shared data exploration.
- Window Embedding - Hosts the visualization interface directly inside a native application window or process.
- Visualization and Analysis - SDK for logging and visualizing multimodal spatial data.
- Data Visualization and Processing - Time-aware visualization stack for multimodal sensor data.
- Robotics Libraries - SDK for logging and visualizing robotics data over time.
