Transcription becomes much more useful when every word has a speaker, timing, delivery details, and conversational context.
2
Speaker diarization still struggles with overlapping speech, short turns, unknown speaker counts, and difficult acoustic conditions.
3
Combining diarization with speech-to-text requires a separate reconciliation step because word timestamps and speaker boundaries often disagree.
Summary
Hervé Bredin argues that speech-to-text alone leaves out information needed to understand conversations. A useful system needs to identify who speaks when, attach speakers to individual words, preserve interruptions and backchannels, and capture details such as pauses, stress, coughing, and laughter. He explains why speaker diarization remains difficult, especially when speakers overlap or recordings come from distant microphones. His demos compare the open source pyannote Community-1 model with pyannoteAI's Precision-2 model, then show how transcription and diarization timestamps can disagree around individual words. A word may sit between two speaker turns, while overlapping speech may produce two speakers but only one transcribed word. Bredin describes pyannoteAI's orchestration layer, including exclusive diarization, as a way to reconcile these outputs without changing the speech-to-text model. The talk is practical and honest about current limits, with error rates varying sharply between clean telephone calls and noisy restaurant recordings.
Speaker labels make ordinary transcripts understandable
Transcription answers what was said, but it does not identify the people speaking. Bredin calls the next step speaker-attributed transcription, where a speaker tag is assigned to each word. That is enough for uses such as meeting summaries, where an action point needs to be assigned to the right person. He also gives examples where speaker identity matters directly, including automatic video dubbing, consistent voices across translated video, and tracking a guest across podcast episodes. Without knowing who said what, the words may be accurate while the conversation remains hard to follow.
Precise timing reveals interruptions and conversational signals
Speaker identity alone does not capture how a conversation unfolds. Bredin says the exact timing of words can reveal that one person interrupted another, or that a short 'Mhm' or 'okay' was a backchannel showing agreement. Timing also preserves pauses, which can convey a speaker's state of mind or change the point they are making. He extends the idea to delivery details such as laughter, coughing, stress, disfluency, and prosody. Stressing different words in 'the dog ate the cake' can alter the meaning, so these details could provide useful input to later language models.
Diarization first finds speech regions, then assigns speaker identities
Bredin describes speaker diarization as answering 'who speaks when.' A system first performs voice activity detection, then divides speech into turns and finds speaker changes, overlap, and short backchannels. It finally assigns each turn to an anonymous speaker such as speaker one or speaker two. The system usually does not know how many speakers are present, and the labels do not need to match real names. Two speakers can swap labels and the diarization remains correct. Unknown speaker counts, overlapping speech, short turns, unequal speaking time, and acoustic conditions all make the task difficult.
Diarization error rate combines several kinds of mistakes
In his phone-call demo, Bredin shows how diarization is evaluated with diarization error rate, or DER. The reference annotation is compared with the model output to identify confusion, false alarms, and misdetections. Confusion assigns speech to the wrong speaker. A false alarm detects speech where the reference has none. A misdetection misses speech, including one speaker during overlap. On the example, the open source Community-1 pipeline produces a 5% diarization error rate, while Precision-2 produces 3%.
Real-world diarization quality depends heavily on the recording
Bredin says state-of-the-art diarization varies sharply by use case. For two-person telephone speech, the best systems can reach 8% diarization error rate in the example he gives. In a noisy restaurant with many people and background sound, the error rate can rise to 41%. He uses this gap to explain why diarization is still an unsolved problem. Performance depends on overlap, background noise, microphone placement, and the structure of the conversation, so a single benchmark number does not describe every deployment.
ASR leaderboard results can hide the difficulty of multi-speaker audio
Bredin compares Nvidia Parakeet's result on the open ASR leaderboard with his own evaluation on the same AMI meeting data. The leaderboard reports an 11.4% word error rate, while his run gets 26%. The difference comes from the microphones used. The leaderboard result uses headset audio, which is closer to single-speaker speech. His result uses the microphone in the middle of the meeting table, where several distant speakers and overlapping voices are captured together. The same model therefore faces a much harder input.
Word-level speaker attribution fails where timestamps disagree
After showing Parakeet word timestamps and Precision-2 speaker regions, Bredin demonstrates why joining the outputs is difficult. A word such as 'Oh' falls between two diarization speech turns, so the system has no obvious speaker to assign it to. Elsewhere, diarization detects two overlapping speakers while transcription produces only one word. Other failures happen when diarization detects speech that ASR does not transcribe, or when ASR outputs words outside the diarized regions. Speaker attribution therefore needs reconciliation rather than a simple timestamp lookup.
Exclusive diarization helps any speech-to-text model handle overlap
Bredin says pyannoteAI's orchestration layer combines Precision-2 diarization with Parakeet transcription and resolves conflicts for the user. In the demo, it interleaves words from two speakers during an overlap and places an interruption correctly. He describes one available part of the approach as exclusive diarization. When two people overlap, the system selects the speaker most likely to be transcribed by the ASR model, which simplifies reconciliation. This step is separate from training the ASR model and is intended to work with different speech-to-text systems, including internally fine-tuned ones.
"When doing speaker attributed transcription, the reason why it might go wrong is either because STT doesn't work great or it's usually the case that they don't generalize very well to multi-speaker recordings."18:45
Who should watch
You are building meeting, call, podcast, or video tools and need speaker labels that remain useful during interruptions and overlap.
Your ASR benchmark looks good on clean audio, but your application uses table microphones, distant speakers, or noisy rooms.
You need to combine a transcription model with diarization and want to understand why assigning speakers by timestamp is unreliable.
