What word error rate does not measure

1. Scope

This note examines word error rate (WER) and its edit-distance variants as instruments for assessing automatic speech recognition (ASR) systems deployed in high-stakes settings. It is a critical review of published work; it does not report new experiments.

Three things are deliberately out of scope. Semantic and task-based ASR metrics are discussed only in §7, as responses to the problems identified here, and are not themselves evaluated. Machine translation metrics (BLEU, chrF, COMET and their relatives) are the subject of a separate note. And this note proposes no alternative methodology: it argues that a gap exists and stops there.

2. What WER is, and what it was built for

For a reference transcript of N words and a system hypothesis aligned to it by minimum edit distance, WER is defined as

WER = (S + D + I) / N

where S, D and I are the counts of substituted, deleted and inserted words. Related measures apply the same alignment to different denominators or units: character error rate operates over characters rather than words, and Morris, Maier and Green (2004) proposed two further measures, match error rate, the proportion of aligned word pairs that are errors, and word information lost, an approximation of the share of word information not communicated. The properties discussed below hold across this family.

Two features of the definition matter for what follows. First, every edit carries unit cost: the metric contains no notion of which words were wrong. Second, WER is a corpus-level ratio, reported as a single figure over a test set.

Neither is a defect. WER was standardised inside benchmark evaluation campaigns to answer a specific question: given two systems and a fixed corpus, which transcribes it more accurately. It answers that question cheaply, reproducibly, and without human judgement. Its dominance is earned. The argument of this note is not that WER is a poor metric, but that system ranking and deployment risk assessment are different questions, and that the properties which make WER good at the first make it structurally unable to answer the second.

It is worth noting that this was never a technical limitation. NIST’s sclite, the reference scoring tool for ASR benchmark evaluations, computes equal-weight word scoring by default but has long offered weighted-word scoring as an alternative, in which individual words carry different weights. The capacity to say that some words matter more than others has been present in the standard toolchain for decades. It is simply not what gets reported.

One preliminary caveat on reported figures. WER is conditional on the corpus it was computed over. Szymański et al. (2020) compared three commercial ASR systems on public benchmarks and on an internal set of real spontaneous call-centre conversations spanning five domains, and found error rates substantially higher than commonly cited results suggest, in both settings and across all domains tested. A published WER is a measurement of a system on a corpus, not a property of the system.

Because every edit costs one, WER treats a deleted hesitation marker and a substituted negation as equivalent events. Two systems with identical WER may differ entirely in which words they transcribe incorrectly, and the metric is silent on that difference by construction.

Kim et al. (2021) demonstrated this directly. Holding WER fixed, they constructed hypothesis sets whose errors were deliberately meaning-damaging and others whose errors were deliberately meaning-preserving, such as reordering adjacent words or inserting articles, and showed that these score identically under WER while diverging under a sentence-embedding distance measure computed from a pre-trained contextual language model. Their conclusion was that not all errors are equal, and that a focus on WER alone will miss the distinction.

The clinical literature supplies the corresponding field measurement. Zhou et al. (2018) annotated 217 clinical documents dictated through speech recognition software at two health systems in the United States. The unedited system output carried an overall error rate of 7.4%, with errors present in 96.3% of documents at that stage. Errors judged clinically significant, meaning those with the potential to affect care, occurred at a rate of approximately one word in 250. After editing by professional medical transcriptionists the overall rate fell to 0.4%, and to 0.3% once reviewed and signed by the dictating physician.

Two quantities are reported there, and they are not interchangeable. The overall error rate and the clinically significant error rate were separately established: the latter required human annotation against a definition of clinical significance, and cannot be derived from the former. Nothing in the study licenses the inference that reducing overall WER reduces the clinically significant fraction proportionally, or at all.

This is the central structural point. For a deployed system, the quantity of interest is the rate of errors that change meaning in a way that affects whatever the transcript feeds: a diagnosis, a legal record, a translation. WER measures the rate of errors of all kinds. The relationship between the two is empirical rather than definitional, and the sections below give reasons to expect it to be unstable.

4. Aggregation: the average is not the user

A corpus-level ratio describes a population of utterances. It does not describe any particular speaker, and it does not decompose unless someone chooses to decompose it.

Koenecke et al. (2020) analysed a corpus of sociolinguistic interviews with white and African American speakers, scoring five commercial ASR systems from major technology providers. They report an average WER of 0.35 for Black speakers against 0.19 for white speakers, with the largest error rates observed for Black men. Within the African American speaker group, error rates rose with a measure of dialect density: speakers exhibiting more features characteristic of African American Vernacular English were transcribed less accurately. The authors attribute the gap primarily to the acoustic models rather than the language models.

The specific finding concerns United States English and a specific vernacular, and should not be extended to other speaker populations without evidence. The transferable claim is narrower and structural: a single reported WER is compatible with a distribution containing subgroups at roughly double that rate, and nothing in the metric surfaces this. Whether analogous gaps exist for second-language speakers, for regional varieties, or for the low-resource languages where evaluation data is thinnest is an empirical question, and one that a single aggregate figure is constitutionally unable to raise.

This compounds §3. A system may be safe on average and unsafe for a subgroup, and severity may itself be distributed unevenly across that same subgroup. Neither dimension is visible in the headline number.

5. Errors that are not edits

WER presumes that the hypothesis is an attempt at the reference, corrupted by substitution, deletion and insertion. Content that was never spoken does not fit that model cleanly: it is scored as a run of insertions, which places it on the same scale as scattered word-level slips.

Koenecke et al. (2024) evaluated OpenAI’s Whisper on audio recordings and found that roughly 1% of transcriptions contained entire hallucinated phrases or sentences with no counterpart in the underlying audio. Analysing the content of those hallucinations thematically, they found that 38% contained explicit harms, including fabricated violent content, invented associations between people and events, and spurious claims of authority. The likelihood of hallucination was associated with longer non-vocal periods in the recording, and was elevated for speakers with aphasia relative to a control group. The authors note that OpenAI has since revised the model and that hallucination rates have fallen.

Three observations follow. A fabricated sentence and an equivalent number of scattered substitution errors are commensurable under WER but not under any reasonable account of harm, since fabricated content is fluent, plausible, and carries no signal of its own unreliability. The failure is speaker-dependent in the same way §4 describes, and disfluent speech (hesitation, pausing, disordered production) appears to be a risk factor, which is a poor property for a system intended for interpreted conversation, distressed speakers, or clinical settings. And this failure mode is specific to the current generation of large sequence-to-sequence recognisers rather than a general property of ASR, which means the evaluation literature built around earlier architectures does not automatically cover it.

6. Stage metrics do not compose

Where ASR is a component rather than a product, its metric is a proxy for something else: the quality of whatever the transcript is used to do. The proxy is imperfect, and can invert.

Wang, Acero and Chelba (2003) tested the assumption that better recognition accuracy yields better spoken language understanding, holding the understanding component fixed. Using a language model trained to optimise understanding accuracy rather than transcription accuracy, they obtained a system with word error rate 46% higher than a trigram baseline, while reducing slot understanding error by as much as 17%. The result is more than two decades old and the architectures have changed entirely, but the structural point survives: WER and downstream task success are distinct objectives, and optimising the first does not entail improving the second.

For any cascaded system, whether recognition feeds translation, summarisation, or record-keeping, this has a direct consequence. Stage-level metrics do not compose into a system-level guarantee. The error rate of the recognition stage constrains what the downstream stage receives, but the relationship between that constraint and the quality of the final output is mediated by which errors occurred and what the downstream component does with them. Reporting stage metrics and reporting system quality are separate exercises.

7. What has been proposed

The problems above are recognised in the literature, and several responses exist.

Semantic distance measures. Kim et al. (2021) proposed scoring the distance between reference and hypothesis in a sentence-embedding space rather than by edit count, and evaluated it against intent recognition, semantic parsing and named entity recognition. Subsequent work by the same group reported that the measure correlated better than WER with user-annotated judgements of output quality across large annotation sets.

Entity and keyword error rates. Restricting the error count to named entities or task-relevant terms is an established practice, and directly targets the observation that some words matter more than others. Kim et al. note that it remains a lexical measure and does not capture meaning damage outside the entity set.

Downstream task evaluation. Measuring the performance of the system that consumes the transcript sidesteps the proxy problem entirely, at the cost of requiring a defined downstream task with its own labelled evaluation data.

Human severity annotation. The clinical work discussed in §3 illustrates the approach: annotators classify errors against a domain-specific definition of significance. It produces the quantity of interest directly, and it is expensive, corpus-specific, and dependent on the availability of qualified annotators.

Weighted scoring. As noted in §2, weighted-word scoring exists in the standard NIST toolchain and has for a long time. Its use requires a weighting scheme, which is a domain judgement rather than a technical artefact.

8. The gap

None of these responses has consolidated into a reporting standard. Semantic measures are used in research and rarely in system documentation; entity error rates are domain-specific and inconsistently defined; downstream evaluation is not portable across deployments; severity annotation is bespoke each time it is performed.

The contrast with machine translation is instructive. There, error-typology frameworks with explicit severity levels, such as the Multidimensional Quality Metrics family applied in the annual WMT human evaluation campaigns, provide a shared vocabulary for describing not merely how much output is wrong but how badly, and what kind of wrong. Nothing of comparable standing exists for speech recognition. An operator who wants to state how often their system produces an error capable of causing harm, in a form another operator could replicate, has no established method to follow.

That is the state of the field as we read it. What follows from it for anyone building on ASR in a high-stakes setting, and what we intend to do about it, is a separate question, and one we will take up when we have something measured rather than something argued.

Sources: A. C. Morris, V. Maier & P. Green, “From WER and RIL to MER and WIL,” Interspeech 2004. Y.-Y. Wang, A. Acero & C. Chelba, “Is word error rate a good indicator for spoken language understanding accuracy,” IEEE ASRU 2003. L. Zhou et al., “Analysis of Errors in Dictated Clinical Documents Assisted by Speech Recognition Software and Professional Transcriptionists,” JAMA Network Open 1(3), 2018. A. Koenecke et al., “Racial disparities in automated speech recognition,” PNAS 117(14), 2020. P. Szymański et al., “WER we are and WER we think we are,” Findings of EMNLP 2020. S. Kim et al., “Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding,” Interspeech 2021, and “Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metric,” 2021. A. Koenecke, A. S. G. Choi, K. X. Mei, H. Schellmann & M. Sloane, “Careless Whisper: Speech-to-Text Hallucination Harms,” ACM FAccT 2024. NIST Speech Recognition Scoring Toolkit (SCTK) documentation.