Hugging Face Daily Papers · July 1, 2026 · 6 min read

RedVox: Safety and Fairness Gaps in Speech Models Across Languages

#model-release #voice #benchmark #safety

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

🌍 Speech AI is going multilingual—but its safety evaluations largely aren't. We present RedVox, a multilingual benchmark for speech model safety and fairness built from real spoken interactions across five languages. Evaluating eight leading models, we find consistent safety and fairness gaps beyond English and show that spoken inputs can further amplify vulnerabilities—highlighting a key blind spot in current speech AI evaluation.\n","updatedAt":"2026-07-01T08:47:49.778Z","author":{"_id":"66309b3833ccd9e68c5d5171","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66309b3833ccd9e68c5d5171/UGt7pZazJHhtwjg0iiyBu.jpeg","fullname":"Sara Papi","name":"spapi","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":14,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9101206064224243},"editors":["spapi"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/66309b3833ccd9e68c5d5171/UGt7pZazJHhtwjg0iiyBu.jpeg"],"reactions":[],"isReport":false}},{"id":"6a45c365f2a6f053303f816a","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":372,"isUserFollowing":false},"createdAt":"2026-07-02T01:48:21.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models](https://huggingface.co/papers/2606.03793) (2026)\n* [SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio Language Models under Code-Switched Speech](https://huggingface.co/papers/2606.06037) (2026)\n* [RealityTest: How People Probe AI Identity and Whether Models Disclose It](https://huggingface.co/papers/2606.00168) (2026)\n* [Yuvion LLM: An Adversarially-Aware Large Language Model for Content And AI Safety](https://huggingface.co/papers/2606.27632) (2026)\n* [Ethical and Technical Limits of Deepfake Speech Datasets](https://huggingface.co/papers/2606.10911) (2026)\n* [MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models](https://huggingface.co/papers/2606.07706) (2026)\n* [IndicGuard: A Multilingual Safety Guard Model and Dataset for Indic Languages](https://huggingface.co/papers/2606.22841) (2026)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"This is an automated message from the <a href=\"https://huggingface.co/librarian-bots\">Librarian Bot</a>. I found the following papers similar to this paper. \nThe following papers were recommended by the Semantic Scholar API \n<ul>\n<li><a href=\"https://huggingface.co/papers/2606.03793\">Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2606.06037\">SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio Language Models under Code-Switched Speech</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2606.00168\">RealityTest: How People Probe AI Identity and Whether Models Disclose It</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2606.27632\">Yuvion LLM: An Adversarially-Aware Large Language Model for Content And AI Safety</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2606.10911\">Ethical and Technical Limits of Deepfake Speech Datasets</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2606.07706\">MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2606.22841\">IndicGuard: A Multilingual Safety Guard Model and Dataset for Indic Languages</a> (2026)</li>\n</ul>\n Please give a thumbs up to this comment if you found it helpful!\n If you want recommendations for any Paper on Hugging Face checkout <a href=\"https://huggingface.co/spaces/librarian-bots/recommend_similar_papers\">this</a> Space\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: <code>@librarian-bot recommend</code>\n","updatedAt":"2026-07-02T01:48:21.475Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":372,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7139589190483093},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.26968","authors":[{"_id":"6a4263ab0dbbc53604b66c23","name":"Beatrice Savoldi","hidden":false},{"_id":"6a4263ab0dbbc53604b66c24","user":{"_id":"66309b3833ccd9e68c5d5171","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66309b3833ccd9e68c5d5171/UGt7pZazJHhtwjg0iiyBu.jpeg","isPro":false,"fullname":"Sara Papi","user":"spapi","type":"user","name":"spapi"},"name":"Sara Papi","status":"claimed_verified","statusLastChangedAt":"2026-06-29T13:21:28.489Z","hidden":false},{"_id":"6a4263ab0dbbc53604b66c25","name":"Wafa Aissa","hidden":false},{"_id":"6a4263ab0dbbc53604b66c26","name":"Matteo Negri","hidden":false},{"_id":"6a4263ab0dbbc53604b66c27","name":"Luisa Bentivogli","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/66309b3833ccd9e68c5d5171/IT5ZKeh3e_bQT_klgitnj.png"],"publishedAt":"2026-06-25T00:00:00.000Z","submittedOnDailyAt":"2026-07-01T00:00:00.000Z","title":"RedVox: Safety and Fairness Gaps in Speech Models Across Languages","submittedOnDailyBy":{"_id":"66309b3833ccd9e68c5d5171","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66309b3833ccd9e68c5d5171/UGt7pZazJHhtwjg0iiyBu.jpeg","isPro":false,"fullname":"Sara Papi","user":"spapi","type":"user","name":"spapi"},"summary":"Speech-capable models are increasingly deployed in real-world applications across languages. Yet their safety and fairness beyond English settings and under naturalistic conditions remain understudied. We survey safety reporting practices across state-of-the-art speech model releases, finding that only 8% document any multilingual analysis. To address this gap, we introduce RedVox, a multilingual safety and fairness benchmark for audio and speech built on real voices, covering unsafe and unfair stereotypical requests across five languages (English, French, Italian, Spanish, and German). Evaluating eight state-of-the-art models, we find that vulnerabilities persist even under non-adversarial conditions, worsen in non-English languages, and are amplified when the request comes from a spoken input. Finally, by surveying the participants who contributed to RedVox, we document the unique personal and privacy challenges of collecting speech data with human participants, pointing to broader sociotechnical challenges in naturalistic speech safety research.","upvotes":12,"discussionId":"6a4263ab0dbbc53604b66c28","ai_summary":"Multilingual safety and fairness benchmark for speech models reveals persistent vulnerabilities across languages and naturalistic conditions.","ai_keywords":["speech-capable models","multilingual safety","fairness benchmark","audio","speech models","naturalistic conditions","vulnerable populations"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","organization":{"_id":"662e2e54ac05b4f7c2b35b47","name":"FBK-MT","fullname":"FBK-MT","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/63b58c53bda8d44adf3092de/ERRcE-4zQH6NM-i7hh8Ro.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"66309b3833ccd9e68c5d5171","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66309b3833ccd9e68c5d5171/UGt7pZazJHhtwjg0iiyBu.jpeg","isPro":false,"fullname":"Sara Papi","user":"spapi","type":"user"},{"_id":"662f5cbf42475b360dd6d1e0","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uSXdcgU4zfKP8ysdAPn0W.jpeg","isPro":false,"fullname":"Roldano Cattoni","user":"cattoni","type":"user"},{"_id":"66350219843f549fdac86347","avatarUrl":"/avatars/1ad7aa4e8e6d80c5d50bf4de502f11a4.svg","isPro":false,"fullname":"Matteo Negri","user":"MNegri","type":"user"},{"_id":"65a42f1a8df9302d1519bc54","avatarUrl":"/avatars/2d39ec5c5a87a8f89c6c0f7a2af3cdcc.svg","isPro":false,"fullname":"Luca","user":"lucacorbucci","type":"user"},{"_id":"63b58c53bda8d44adf3092de","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63b58c53bda8d44adf3092de/7Qs7MmPHmQtZ7Hno5n6-P.jpeg","isPro":false,"fullname":"Beatrice Savoldi","user":"BSavoldi","type":"user"},{"_id":"668961b8b32d04463ba964b8","avatarUrl":"/avatars/2249a1fe47c5fb71a4377e0ec45f62da.svg","isPro":false,"fullname":"Yiu Chung Leung","user":"ycleungaj","type":"user"},{"_id":"63c3ffc1c7d7f4c63a52b297","avatarUrl":"/avatars/fb985e9f0d75e47ed15b83c60dc75024.svg","isPro":false,"fullname":"Lina Conti","user":"lina-conti","type":"user"},{"_id":"662eb48f5ade870b60db4380","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ozHyP-GBIAEhR-l-TkBFb.jpeg","isPro":false,"fullname":"Luisa Bentivogli","user":"lubentivogli","type":"user"},{"_id":"60d494e250c47659f83f5cd0","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1670856721262-60d494e250c47659f83f5cd0.png","isPro":false,"fullname":"Giuseppe Attanasio","user":"g8a9","type":"user"},{"_id":"681c62df67484b747c2d6407","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/681c62df67484b747c2d6407/Qs-TOqeyqBMB2XHVhpQNG.jpeg","isPro":false,"fullname":"Orfeas Menis Mastromichalakis","user":"menorf","type":"user"},{"_id":"662e58f32b1b529a43c8b2f4","avatarUrl":"/avatars/bd26a7b26596882c6bc8a7375ff42cac.svg","isPro":false,"fullname":"Marco Gaido","user":"mgaido91","type":"user"},{"_id":"6a2da6c8ca070ee12c6e396c","avatarUrl":"/avatars/0355287dcabaa67dbc7f0b10b87451f9.svg","isPro":false,"fullname":"Joe Mama","user":"JoeMama123123123","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"662e2e54ac05b4f7c2b35b47","name":"FBK-MT","fullname":"FBK-MT","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/63b58c53bda8d44adf3092de/ERRcE-4zQH6NM-i7hh8Ro.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.26968.md","query":{}}">

Papers

arxiv:2606.26968

RedVox: Safety and Fairness Gaps in Speech Models Across Languages

Published on Jun 25

· Submitted by

Sara Papi on Jul 1

FBK-MT

Upvote

Authors:

Sara Papi ,

Abstract

Multilingual safety and fairness benchmark for speech models reveals persistent vulnerabilities across languages and naturalistic conditions.

Generated by Qwen/Qwen2.5-Coder-32B-Instruct

Speech-capable models are increasingly deployed in real-world applications across languages. Yet their safety and fairness beyond English settings and under naturalistic conditions remain understudied. We survey safety reporting practices across state-of-the-art speech model releases, finding that only 8% document any multilingual analysis. To address this gap, we introduce RedVox, a multilingual safety and fairness benchmark for audio and speech built on real voices, covering unsafe and unfair stereotypical requests across five languages (English, French, Italian, Spanish, and German). Evaluating eight state-of-the-art models, we find that vulnerabilities persist even under non-adversarial conditions, worsen in non-English languages, and are amplified when the request comes from a spoken input. Finally, by surveying the participants who contributed to RedVox, we document the unique personal and privacy challenges of collecting speech data with human participants, pointing to broader sociotechnical challenges in naturalistic speech safety research.

View arXiv page View PDF Add to collection