Neutral speech is the usual basis for automated TTS evaluation, but expressive output does not fit that assumption. Standard listening corpora, the ITU-T P.800 methodology, and neural scoring models trained on those datasets largely represent clear, fluent, relatively unexpressive reading. That foundation works for many TTS applications. It becomes less reliable for emotional and expressive speech. Applying a neutral-calibrated MOS model without accounting for that difference can lead teams to misinterpret their results.
This post explains where the Podonos scoring model remains useful for expressive speech, where confidence intervals become wider, and how to pair automated scores with human evaluation when expressivity is central.
Why expressive speech is harder to score
Neutral speech occupies a comparatively narrow target range. A successful neutral utterance can sound right across differences in speaking pace and energy. Listeners also share fairly consistent expectations for neutral TTS: clarity, fluency, and a human quality. Because those expectations overlap, models trained on neutral listener ratings generalize reasonably well across many neutral TTS outputs.
Expressive speech covers a wider and more varied target. Anger, frustration, warmth, excitement, and somber gravity represent five different emotional registers. Each involves its own combination of pitch movement, energy, rate, and phonation, as well as its own standard for convincing delivery. Frustration may sound appropriate in one cultural setting but excessive in another. Warmth can arise through several distinct acoustic patterns. The expressive target is therefore not only broader than the neutral one, but also more dependent on context.
MOS confidence intervals are consequently wider for expressive speech. The issue is not necessarily lower model accuracy, but less stable ground truth. Ten listeners are likely to show more disagreement when rating the naturalness of an expressive utterance than a comparable neutral one. That listener variation appears in the automated result as a wider confidence interval.
Where automated scoring remains reliable
Podonos scores tend to track human listener ratings well for mild positive expression, conversational warmth, gentle emphasis, and the soft enthusiasm common in guide-style or tutorial speech. These forms remain sufficiently near neutral on the expressivity spectrum for the calibration data to cover them reasonably well.
Reliability declines with stronger emotional signals, including pronounced excitement, anger, alarm, irony, sarcasm, uncertainty, and culturally specific registers. In these cases, do not use aggregate MOS as the main quality gate without a human listen. The score still provides information, but equating a 3.8 in excited speech with a 3.8 in neutral speech implies more precision than the result supports.
For expressive speech, pronunciation and naturalness usually remain more dependable than prosody. Phoneme accuracy is not strongly tied to emotional content, so the pronunciation model is largely invariant to expressivity. Waveform naturalness mainly reflects vocoder and signal quality, which are also relatively independent of expression. Prosody departs most from the neutral distribution, and that difference appears specifically as wider confidence intervals for the prosody sub-score.
Building a hybrid evaluation plan for expressive TTS
For expressive TTS, the practical choice is to combine automated scores with focused human evaluation, not to choose one over the other.
Keep automated scoring in the frequent development loop: run it at every training checkpoint and architecture experiment. Its MOS gate can catch broad regressions in audio quality and basic naturalness, including expressive output, even when confidence intervals widen. A real change from 3.8 to 3.2 remains meaningful despite that added variance.
Reserve human listeners for qualitative expressivity: is an excited voice convincingly excited or simply louder, and does a warm voice feel warm in its intended context? Automated scoring is not built to answer these questions. Evaluate each emotional register used by the product at every major model milestone.
For expressive speech, annotation design has a major effect on human-evaluation results. Asking listeners to rate overall voice quality combines several dimensions and can create noisy data. More useful tasks separate the questions: does the emotion come through, does the voice sound natural in that emotional state, and does the delivery fit the text? These responses are more actionable than one overall quality score.
Measuring emotion category consistency separately
Aggregate MOS does not measure emotion category consistency in expressive TTS: whether listeners hear the intended emotion, regardless of whether the voice sounds natural. A voice may receive a strong MOS while failing to communicate its target emotion. This requires a separate design, such as a forced-choice or rating task asking listeners to identify or assess the output's emotional content.
Podonos does not currently include emotion category consistency. Creating a well-calibrated emotion-recognition task would require careful annotation and multi-cultural validation beyond the current API scope. This limitation is known. Teams that do not separately assess emotion conveyance leave a major quality dimension unchecked. Automated naturalness and prosody scores matter, but they do not cover expressive TTS evaluation by themselves.
Practical advice on calibration data
Teams building expressive TTS can improve automated scoring by collecting targeted human ratings for their own expressive outputs. Those ratings support a custom threshold that reflects the wider variance of expressive speech and provide ground truth for checking whether automated scores follow the application's priorities. A human-annotated batch once per quarter cannot replace the daily automated loop, but it can supply the calibration anchor that makes that loop more trustworthy.
The main takeaway is that MOS remains useful for expressive TTS, though it is noisier and prosody is affected most. Rather than dropping automated scoring, pair it with human evaluation designed for expressivity. General naturalness ratings alone are not enough.