SenseVoice

SenseVoice is a multilingual speech large language model designed for audio transcription, speaker diarization, and emotion recognition. It functions as an automatic speech recognition system that converts spoken audio into text across multiple languages.

The system distinguishes itself by integrating acoustic event detection and speech emotion recognition, allowing it to identify non-speech sounds, such as laughter or applause, and discrete emotional states. It also includes a framework for speaker diarization to track and label different speakers within a single recording.

The project's capabilities extend to speech synthesis, including expressive text-to-speech, zero-shot speaker identity cloning, and voice interpolation. It further provides tools for speech model fine-tuning to optimize performance for specific domains or rare languages.

Features

Multilingual Transcription - Converts spoken audio into text across multiple languages using low-latency processing and automatic language detection.

Automatic Speech Recognition - Functions as a multilingual automatic speech recognition system converting spoken audio to text.

Speech Synthesis - Generates natural sounding human speech across multiple languages with controllable pitch, rate, and emotional tone.

Zero-Shot Voice Cloning - Features zero-shot voice cloning to synthesize target voices from short reference clips without retraining.

Spoken Language Detection - Detects spoken languages in audio samples to route data to the appropriate transcription pipelines.

Multilingual Speech-to-Text - Provides a large language model for low-latency transcription across multiple languages.

Speaker Diarization - Labels different speakers in a recording using voice activity detection and speaker modeling.

Audio Emotion Classifiers - Analyzes acoustic signals to identify discrete emotional states like happiness or anger.

Audio Event Detection - Identifies non-speech acoustic events such as laughter or applause to categorize audio environments.

End-to-End Pipelines - Implements an end-to-end pipeline mapping raw audio waveforms directly to text tokens.

Inference Latency Optimizers - Utilizes a streaming architecture for latency-optimized inference to decode speech tokens in real-time.

Speech Model Fine-Tuning - Offers frameworks for adapting speech models to specific business scenarios or rare language samples.

Speech Model Training - Provides tools for fine-tuning speech models for specific business domains or rare languages.

Acoustic-Prosodic Embeddings - Implements acoustic-prosodic embeddings to encode emotional states and non-speech events as tokens.

Shared Acoustic Encoders - Uses a multilingual shared encoder to extract acoustic features across diverse languages and events.

Voice Cloning - Replicates specific human vocal characteristics from audio samples using zero-shot generation or fine-tuning.

Audio and Speech Models - Multitask foundation model for speech, emotion, and audio events.

FunAudioLLMSenseVoice

Features

Star history