Personalized fine-tuning of Whisper achieves 9.7% WER on dysarthric speech
Researchers adapted Whisper to a single dysarthric speaker using up to 100.8 hours of read speech and user corrections collected via a mobile app, reducing word error rate from a high baseline to 9.7%. Fine-tuning outperformed LoRA adaptation and the Qwen3-ASR foundation model in this personalized setting. The study demonstrates that speaker-specific fine-tuning of foundation ASR models can reach practical deployment quality for dysarthric users.
Related guides (1)
Related events (8)
Case study compares human and ASR system performance on Dutch dysarthric speech recognition
A new arXiv preprint compares human listeners against three off-the-shelf ASR systems (Whisper-large-V3, Google Chirp 3, and Omnilingual) on recognizing continuous Dutch speech from a single speaker with severe dysarthria. Both humans and ASR systems exceeded 70% WER on average, confirming the extreme difficulty of dysarthric speech recognition. Fine-tuning on dysarthric speech substantially reduced WER, with personalized models outperforming human listeners, though WER remained above 23%. The study highlights the need for personalized ASR approaches for dysarthric speakers.
Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers
This Hugging Face blog post provides a practical guide for fine-tuning OpenAI's Whisper model for multilingual automatic speech recognition using the Transformers library. It covers dataset preparation, training configuration, and evaluation using the Word Error Rate metric. The post targets practitioners seeking to adapt Whisper to low-resource or domain-specific languages.
Introducing Whisper
OpenAI introduced Whisper, an open-source automatic speech recognition (ASR) system trained on 680,000 hours of multilingual and multitask supervised data collected from the web. The model demonstrates strong robustness to accents, background noise, and technical language, approaching human-level accuracy in English transcription. Whisper supports transcription in multiple languages as well as translation to English, and the weights and inference code were released publicly.
Speech-based dementia screening using Whisper embeddings to compensate for nonverbal subtest omissions
Researchers present a speech-based evaluation system for the German Syndrom-Kurz-Test dementia screening battery, combining transcript-derived scores with Whisper embeddings to reduce transcription scoring errors. The system also approximates expert overall ratings even when motor (nonverbal) subtests are omitted, addressing a key accessibility limitation of speech-only assessment. Models show strong correlation with expert ratings and effective discrimination between cognitive status groups.
Blazingly Fast Whisper Transcriptions with Inference Endpoints
Hugging Face published a blog post detailing optimized Whisper speech-to-text transcription deployments via their Inference Endpoints service. The post covers performance improvements using faster-whisper or similar optimized backends to achieve significantly reduced transcription latency. This is positioned as a practical deployment guide for production speech recognition workloads.
Thaka Wins KSAA-2026 Arabic Speech Diacritization Task with Regularized Fine-Tuning of CATT-Whisper
The Thaka team describes their winning system for Task 2 of the KSAA-2026 Shared Task on Arabic Speech Dictation with Automatic Diacritization, which requires producing fully diacritized Arabic text from speech audio and undiacritized transcripts. Their approach fine-tunes CATT-Whisper, a multimodal model combining a CATT text encoder with a frozen Whisper speech encoder, under severe data constraints (2,327 training samples, no external data). Key techniques include R-Drop consistency regularization, Optuna-optimized hyperparameters with high weight decay, Focal Loss, and Monte Carlo Dropout inference averaging over 200 stochastic forward passes across four checkpoints. The system achieves 23.26% WER on the primary metric, placing first among all participants.
Fine-Tune MMS Adapter Models for Low-Resource ASR
This Hugging Face blog post provides a technical guide for fine-tuning Meta's Massively Multilingual Speech (MMS) adapter models for automatic speech recognition in low-resource languages. It covers the adapter-based fine-tuning approach that allows efficient adaptation of the MMS model to specific languages without full model retraining. The post targets practitioners working on speech recognition for underrepresented languages.
Sparse AutoEncoder steering reduces Whisper hallucination rate by ~5x without fine-tuning
Researchers investigate hallucination detection and mitigation in OpenAI's Whisper ASR model by probing internal encoder representations. They find that both raw activations and Sparse AutoEncoder (SAE) latents encode linearly separable hallucination signals concentrated in deeper layers. SAE-based activation steering reduces hallucination rates from 72.6% to 14.1% (Whisper small) and 86.9% to 27.3% (Whisper large-v3) on non-speech audio, with minimal WER degradation, approaching fine-tuning-level performance without weight updates.