Rhubarb Lip Sync

Rhubarb is an automated lip sync generator and phonetic speech analyzer that converts audio recordings into timed mouth-shape animation data. It identifies sounds and syllables within audio files to map them to specific visual mouth shapes, serving as an animation timing exporter for external character animation software.

The tool utilizes a language-independent phonetic recognizer to process speech regardless of the spoken language. To increase accuracy, it supports dialogue-guided recognition by using external text files to guide the phonetic analysis of specific spoken scripts.

The system covers audio processing across multiple formats and provides a mapping system to link phonetic events to user-defined mouth assets. It exports synchronization data in various formats for integration with external creative tools, including support for Spine, After Effects, and Magix Vegas.

The application emits machine-readable status updates and progress events via a standard error stream for external tool integration.

Features

Lip-Sync Animations - Analyzes audio recordings of speech to produce timed mouth-shape animation data using phonetic recognition.

Automated Lip Sync Generators - Automatically converts audio recordings into timed mouth-shape animation data for character lip synchronization.

Phonetic Speech Analyzers - A system that identifies sounds and syllables in audio files to map them to specific visual mouth shapes.

Language-Agnostic Speech Recognition - Identifies universal speech sounds and syllables to generate lip sync data regardless of the spoken language.

Phonetic Audio Analysis - Converts audio waveforms into a sequence of timed phonetic markers to determine required mouth shapes.

Phonetic Text Processors - Uses provided dialogue scripts to correct phonetic recognition errors and improve animation accuracy.

Speech Recognition - Processes audio using a phonetic recognizer that identifies sounds and syllables independently of any specific language.

Dialogue-Guided Recognition - Uses external text files to improve phonetic recognition and handle unknown word pronunciations during analysis.

Animation Data Formats - Outputs mouth-shape timing in various structured data formats to ensure compatibility with different animation software.

Animation Pipelines - Generates and exports mouth-shape timing data into formats compatible with various external animation software tools.

Audio-Driven Animation Engines - Uses provided text files to improve the accuracy of phonetic recognition and mouth-shape synchronization for specific scripts.

Script-Based Accuracy Improvements - Uses external text files to guide speech recognition and produce more reliable animation results based on specific dialogue.

Text-Guided Recognition - Uses external text files to guide phonetic recognition and improve the accuracy of mouth-shape synchronization.

After Effects Integrations - Generates automated mouth animations specifically formatted for use within Adobe After Effects via script interfaces.

Spine Animation Integrations - Analyzes audio events to create mouth-shape animations and exports them as compatible data files for Spine.

Animation Exporters - Exports synchronized speech timing data into portable formats compatible with external animation software.

Animation Software Integrations - Exports compatible data files or uses plugins to move animation data into external creative production tools.

Phonetic Asset Mappings - Links recognized phonetic events to specific user-defined mouth assets through a configurable mapping system.

Mouth Shape Customizations - Matches automated phonetic timing data to specific visual assets to fit the unique design of a character.

DanielSWolfrhubarb-lip-sync

Features

Star history