NVIDIA/personaplex
Personaplex
Personaplex is an LLM speech-to-speech framework and conversational AI persona engine designed for real-time voice interfaces. It provides a system for defining AI identities and vocal characteristics through a combination of text-based role prompts and audio reference files.
The project features a real-time AI voice interface that supports full-duplex human-AI dialogue, enabling multiple parties to speak and listen simultaneously via bidirectional audio streaming. It includes a GPU-accelerated audio processor and a speech-to-speech pipeline to facilitate low-latency conversations.
The framework incorporates resource management tools, such as CPU model layer offloading to move components between video and system memory. It also supports offline audio processing for asynchronous generation and session routing to direct clients to specific worker instances.
Features
- Real-Time Conversational AI Frameworks - Provides a full-duplex speech-to-speech framework and web interface for live human-AI dialogue.
- Agent Persona Configurations - Provides a system for customizing the identity and speaking style of AI agents.
- Realtime Voice Conversation Facilitators - Facilitates low-latency, two-way spoken interactions with AI models for real-time conversation.
- Persona Identity Definitions - Defines AI identity, knowledge, and speaking style through a combination of text prompts and audio conditioning.
- Few-Shot Persona Conditioning - Shapes AI identity and speaking style by injecting role-specific prompts and context blocks.
- Full-Duplex Multimodal Interaction - Enables real-time conversational systems where multiple parties can speak and listen simultaneously.
- Persona Management Systems - Provides a dedicated engine for creating and managing AI identities and vocal characteristics.
- Unified Speech Pipelines - Integrates transcription and synthesis into a unified pipeline for end-to-end speech-to-speech processing.
- Speech-to-Speech Frameworks - Implements a comprehensive framework for real-time bidirectional voice conversations using LLMs.
- Voice Cloning - Provides capabilities to replicate specific human vocal characteristics using reference audio samples.
- Full-Duplex Conversational Streams - Implements bidirectional real-time audio streaming to enable simultaneous speaking and listening.
- Vocal Persona Configuration - Allows the definition of AI speaker personality and vocal characteristics using role prompts and audio conditioning.
- Voice Interfaces - Provides a server and web-based voice interface for low-latency, full-duplex human-AI interaction.
- Audio Processing - Supports offline processing of audio files to generate output audio of the same duration.
- Large Model Optimizations - Optimizes large model execution on limited hardware through device mapping and memory offloading.
- Weight Offloading - Implements memory-constrained execution by moving model layers between video and system memory to support larger workloads.
- Audio Generation and Processing - Provides offline tools for synthesizing and transcribing audio files for evaluation and analysis.
- Model Layer Offloading - Enables running large AI models on limited hardware by offloading layers between video and system memory.
- Audio Acceleration - Ships a GPU-accelerated audio processor to handle high-performance speech synthesis and offloading.
