Advanced Zero-Shot Text-to-Speech
for Background Removal and Preservation
with Controllable Masked Speech Prediction

The acoustic background plays a crucial role in natural conversation. It provides context and helps listeners understand the environment, but a strong background makes it difficult for listeners to understand spoken words. The appropriate handling of these backgrounds is situation-dependent: Although it may be necessary to remove background to ensure speech clarity, preserving the background is sometimes crucial to maintaining the contextual integrity of the speech. Despite recent advancements in zero-shot Text-to-Speech technologies, current systems often struggle with speech prompts containing backgrounds. To address these challenges, we propose a Controllable Masked Speech Prediction strategy coupled with a dual-speaker encoder, utilizing a task-related control signal to guide the prediction of dual background removal and preservation targets. Experimental results demonstrate that our approach enables precise control over the removal or preservation of background across various acoustic conditions and exhibits strong generalization capabilities in unseen scenarios.

Scenario: The type of background contained in the input speech prompt
Prompt: The input single-speaker speech of target speaker, which may contains environmental noise, reverberation or interfering speakers
Ours (BR): Our proposed model with CMSP strategy and Dual speaker encoder in background removal mode
Ours (BP): Our proposed model with CMSP strategy and Dual speaker encoder in background preservation mode
Baseline1: The Voicebox model.
Baseline2: The Voicebox model with an extra Speech Enhancement Model
Baseline3: The Voicebox model with MSD strategy
Baseline4: The VALL-E model (open-sourced by https://huggingface.co/amphion/valle_libritts)
All these models are trained on LibriTTS dataset