Common Voice

SN-WER: Script-Normalized Word Error Rate for Multi-Script Indic ASR Evaluation

Researchers propose Script-Normalized WER (SN-WER), a training-free evaluation metric that transliterates ASR reference and hypothesis text into a canonical script before computing WER, addressing overestimation of errors caused by script mismatches in multilingual settings. Evaluated across 5 Indic languages, 2 datasets, and 3 ASR models, SN-WER reduces inflated model performance gaps by up to 12% on curated FLEURS data and attenuates romanization-induced WER inflation by 67% in controlled tests. The metric maintains near-identical sensitivity to genuine semantic errors (ΔSN-WER/ΔWER ≈ 1.09) and shows robustness to transliterator choice with token-collision rates below 0.1%. The authors recommend SN-WER as a companion metric to WER and CER, particularly for pipelines feeding downstream search, indexing, or multilingual LLM applications.