MOS is designed to estimate the perceived quality of synthetic speech, approximating how a listener panel would rate it. That makes automated MOS useful for spotting regressions between training checkpoints, comparing architectures on the same test set, and maintaining an affordable quality signal for every model update. For most general-purpose TTS evaluation, aggregate MOS is the appropriate primary metric.
Some TTS work calls for more than one aggregate score: multi-speaker and voice cloning models, applications with domain-specific vocabulary, and code-switching or multilingual output. In each case, the evaluation question extends beyond what aggregate MOS is built to answer. Here is why, along with the signals to add to automated scoring.
Multi-speaker and voice cloning
For multi-speaker and voice cloning systems, aggregate MOS blends two separate quality dimensions with different failure patterns and remedies. One is the synthesized speech's general naturalness and prosody. The other is speaker similarity, meaning whether the output sounds like the intended target speaker.
A system may generate natural speech with good prosody yet fail to resemble the target speaker. Its naturalness and prosody scores can be strong even though it misses the model's central purpose. The reverse can also happen: the output may closely match the target speaker while sounding stilted and full of artifacts. These cases call for different fixes. The first points to speaker conditioning or the voice encoder; the second points to the acoustic model or vocoder.
Speaker similarity needs its own task design. A common method is SMOS (Speaker similarity MOS), in which listeners hear a reference recording of the target speaker and judge how closely the synthesized voice matches it. That judgment differs from overall quality and yields a different score distribution. MOS cannot be used to derive SMOS, nor can SMOS be inferred from MOS.
Automated speaker similarity can use speaker verification models. Generate speaker embeddings for the target reference and synthesized output with a model such as ECAPA-TDNN or x-vector, then calculate cosine similarity. The result is a continuous automated measure comparable in form to Podonos MOS scoring. However, speaker verification models are calibrated on real speech, so scores for synthesized speech are less dependable than scores for real speech. Treat automated speaker similarity as a relative measure between model versions, not an absolute quality gate.
Domain-specific vocabulary
Typical TTS test sets use general English text, including news, audiobooks, and sentence lists covering a wide phonemic range. Strong performance on those sets does not guarantee correct pronunciation of the vocabulary specific to your application.
Medical systems must handle drug names, anatomical terms, and clinical procedure names absent from standard training corpora. Technical documentation brings acronyms, programming language identifiers, and compound technical terms. Legal text contains Latin phrases and terms of art with conventional pronunciations that may not follow phonetic rules. General MOS on a standard test set will miss the pronunciation errors that affect users in each of these domains.
Use a targeted pronunciation audit for domain vocabulary. Create a test set containing the terms your application will encounter, synthesize them, and compare pronunciation with the expected forms. WER-based intelligibility testing with an ASR model is one option. Expert human review is another, and it is more dependable in high-stakes domains such as medical and legal applications because ASR models can share the pronunciation biases of the TTS model.
Standard-test-set MOS remains useful for monitoring overall changes between checkpoints. Before deployment in a specialized application, pair it with vocabulary tests for each relevant domain. Those tests can run less often than MOS checkpoint evaluations because errors on particular terms usually persist across checkpoints instead of appearing and disappearing with training updates.
Code-switching and multilingual output
Code-switching means alternating between two languages in one utterance, a common feature of bilingual speech. It can occur in product names, such as Spanish brand names read by an English TTS system in marketing copy, as well as in proper nouns and quoted non-primary-language phrases. An MOS model calibrated only on single-language output will produce unreliable scores for code-switched text.
Two issues are involved. The scoring model may not have been calibrated with code-switched audio, so its result is an extrapolation from the calibration distribution rather than a validated prediction. Also, code-switched speech has different perceptual expectations from single-language speech. A bilingual listener judging code-switched TTS applies a different standard from a monolingual listener judging single-language TTS, and a model trained on monolingual ratings will not represent that distinction.
For multilingual and code-switching work, evaluate each language segment separately and use targeted human review for the transitions. Automated MOS on the monolingual portions provides a valid quality signal for those segments. The language changes themselves need bilingual human listeners to judge whether the switch's phonology and prosody sound natural.
What aggregate MOS is designed to cover
The boundary is straightforward: MOS assesses naturalness, prosody, and pronunciation for speech within the scoring model's calibration distribution. It does not assess speaker identity preservation, pronunciation accuracy for specialized domain vocabulary, or the quality of language transitions in code-switched speech. These dimensions matter, but each calls for an evaluation method built around it.
The point is not that automated MOS is inadequate or unreliable. It is effective for the qualities it was designed to measure, provided teams identify the other dimensions their application needs. In a simple single-language, single-speaker TTS application using general vocabulary, aggregate MOS covers the essential quality dimensions well. Multi-speaker, domain-specific, and multilingual applications should layer it with methods for the dimensions MOS does not address.
A tool becomes useful when its limits are understood. Automated MOS estimation is a major improvement over conducting a full human listening study for every iteration. Trouble starts when teams ask it questions outside its design, or treat a PASS on MOS as permission to omit targeted checks for qualities their application depends on.