RedVox: Safety and Fairness Gaps in Speech Models Across Languages

Authors: Beatrice Savoldi, Sara Papi, Wafa Aissa, Matteo Negri, Luisa Bentivogli

arXiv ID: 2606.26968

Problem: Only 8% of speech model releases document any multilingual safety evaluation, leaving dangerous blind spots in safety frameworks as these models are deployed across languages worldwide.

Key Methodology:

Surveyed safety reporting across 38 state-of-the-art speech model releases
Built RedVox, a multilingual (en/fr/it/es/de) safety & fairness benchmark with 3414 entries from 26 natural human voices, covering two request types: Speech (harmful content vocalized) and Audio (harmful content in text + distracting non-speech audio)
Evaluated 8 SOTA models (5 open: Qwen2-Audio, Phi4-Multimodal, Voxtral, Qwen3-Omni, Gemma4; 3 proprietary: Gemini 3.1 Flash-Lite, Gemini Pro-Preview, GPT-realtime-2) using GPT-5.5-as-judge with manual validation

Key Results:

Proprietary models had ≤3.1% unsafe responses; worst open model (Voxtral) hit 21.9% unsafe
Non-English unsafe rate (10.0%) nearly double English (5.1%) - a Δ96% relative increase
Speech input was the most vulnerable modality, reaching 10–44% harmful responses across models
Even non-semantic audio (silence, noise) increased unsafe rates up to +20% vs text-only input
Only 50% of 52 participants consented to public data release; 61.5% found releasing harmful voice recordings uncomfortable; 56.4% felt more personally responsible pronouncing vs writing harmful content

Applied Context: If you're building with speech-capable models, expect audio/speech inputs to systematically bypass safety guardrails more easily than text - and plan for multilingual safety eval as a prerequisite, not an afterthought. The data bottleneck is real: collecting naturalistic harmful speech data faces psychological and privacy barriers that text red teaming does not.