# The End of Awkward AI Transcriptions

Travis Bartley & Myungjong Kim, NVIDIA | AI Engineer World's Fair 2025 | 16:24

Source: https://www.youtube.com/watch?v=aDj9sY2RoG8
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/the-end-of-awkward-ai-transcriptions
Published: 2025-06-03
Tags: deployment, inference, multimodal, voice

## TL;DR
- NVIDIA chooses different speech model architectures for streaming speed, accuracy, and multitask capabilities instead of forcing every use case into one model.
- The Fast Conformer backbone uses 80-millisecond subsampling to reduce memory use and speed up training and inference.
- NVIDIA supports customization at several stages, including acoustic models, language models, punctuation, inverse text normalization, and word boosting.

## Summary
Travis Bartley and Myungjong Kim describe how NVIDIA develops and deploys speech AI models for enterprise applications. Their process starts with four concerns: robustness to noise, coverage across languages and domains, personalization, and the speed-accuracy tradeoff at deployment. CTC and TDT models support fast streaming recognition through Parakeet, while Canary models target higher accuracy and multitask workloads. Both model families use NVIDIA's Fast Conformer architecture, which applies 80-millisecond subsampling. The talk also covers speaker-aware recognition through Softformer, along with voice activity detection, external language models, text normalization, punctuation, capitalization, and speaker diarization. Training relies on open-source and proprietary data, pseudo-labeling, and the NeMo toolkit. NVIDIA deploys trained models through Riva and NIM, with support for cloud, on-premises, edge, and embedded environments. The practical focus is customization for domains such as medicine, food ordering, and contact centers.

## Key ideas
### NVIDIA develops speech models around robustness, coverage, personalization, and deployment needs
[01:11](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=71s)
The team evaluates whether models work in noisy and clean environments, including telephone audio and other environmental contamination. Coverage includes medical, entertainment, and call center domains, plus multilingual speech, dialects, and code switching. Personalization can involve target-speaker recognition, word boosting for uncommon vocabulary, and text normalization with finite-state transducer models. Deployment decisions then weigh speed against accuracy and broad model variety against efficiency. These choices determine which model and pipeline a customer receives.

### CTC, TDT, and attention encoder-decoder models cover different recognition requirements
[02:51](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=171s)
NVIDIA still uses CTC models because non-autoregressive decoding is well suited to high-speed inference and streaming chunks of audio. When accuracy matters more, the team uses RNN-T or NVIDIA's TDT variant, which combines an encoder audio output with an internal language model for autoregressive streaming. Attention encoder-decoder models are useful when streaming and alignment matter less. Transformer decoders can combine speech recognition, translation, timestamp prediction, and language identification through prompt changes.

### Fast Conformer is the shared backbone across NVIDIA's speech model families
[04:23](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=263s)
Through empirical testing, NVIDIA found that the original Conformer could use an additional subsampling step. This changes the conventional 40-millisecond time-step compression to 80 milliseconds. Smaller audio inputs reduce the memory load during training, while the larger time steps help training converge faster with less data. They also support faster inference because the system processes audio in larger chunks. This backbone supports the different decoding approaches used in NVIDIA's speech products.

### Parakeet prioritizes streaming efficiency while Canary prioritizes accuracy and multitask modeling
[05:25](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=325s)
NVIDIA places CTC and TDT models in the Parakeet family for streaming speech recognition. Parakeet covers speech recognition, speech translation, and target-speaker ASR with an emphasis on fast and efficient processing. Canary uses Fast Conformer models for higher accuracy and multitask workloads, with less emphasis on speed even though NVIDIA still works to keep it fast. The team explicitly favors a range of models and coverage for different needs instead of one model that tries to handle every case.

### Softformer connects speaker diarization with speaker-aware speech recognition
[06:41](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=401s)
NVIDIA extends Parakeet to multispeaker and target-speaker scenarios by integrating Softformer, an end-to-end neural diarizer. Softformer connects speaker timestamps from diarization with speaker tokens recognized by the ASR model. The system fuses the ASR encoder embedding and Softformer embedding through a speaker kernel to address who spoke what and when. Optional query audio lets the same architecture perform target-speaker ASR or multispeaker ASR, and the components can run jointly or as a cascade.

### Additional pipeline models improve recognition accuracy and transcript readability
[08:19](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=499s)
The broader toolkit includes voice activity detection for finding speech segments and improving noise robustness. External n-gram language models can improve accuracy and customization. Text normalization and inverse text normalization convert between spoken and written forms. Punctuation and capitalization models make transcripts easier to read. Speaker diarization identifies multiple speakers, with both cascade models and end-to-end models available or planned.

### Training depends on broad data coverage and standard open tooling
[10:56](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=656s)
NVIDIA combines open-source and proprietary data. Open data helps with variety and domain shift, while proprietary data provides higher-quality entity data. The team also uses pseudo-labeling, taking transcripts from strong commercial models and using them in further development. NeMo provides open-source tools for GPU utilization, data bucketing, and high-speed loading through the Lhotse backend. Data is stored on object storage so it can move between clusters, and validation checks language coverage, bias, and domain performance before release.

### Riva and NIM package speech models for scalable deployment and customization
[13:22](https://www.youtube.com/watch?v=aDj9sY2RoG8&t=802s)
Trained models are deployed through NVIDIA Riva and NVIDIA NIM. Tensor optimizations and the Triton Inference Server support low-latency gRPC streaming and high-throughput offline workloads. Riva is containerized and can run on premises, in clouds, at the edge, or on embedded platforms. NIM provides prebuilt containers, standard APIs, and optimized inference engines. Customers can fine-tune acoustic models, external language models, punctuation, and inverse text normalization, and can use word boosting for product names, jargon, and domain-specific terms.

## Notable quotes
- "We focus on variety and coverage rather than unifying and keeping everything under one hood." (06:41)
- "We really focus on the fundamentals when it comes to data development." (10:56)
- "The NVIDIA Riva is fully containerized and it can easily scale to hundreds of the par streams." (13:47)
- "NVIDIA Riva reverses the customization features at every stage." (14:34)

## Tools & references mentioned
- NVIDIA
- NVIDIA Riva
- NVIDIA NIM
- Parakeet
- Canary
- Fast Conformer
- Softformer
- NeMo
- Triton Inference Server
- Hugging Face Open ASR leaderboard
- CTC
- TDT
- RNN-T
- Whisper
- speech translation
- voice activity detection
- text normalization
- inverse text normalization
- speaker diarization

## Who should watch
- You are choosing between streaming ASR, high-accuracy recognition, or a multitask speech model and need to understand the tradeoffs described by NVIDIA.
- Your application needs domain vocabulary, speaker-aware recognition, multilingual coverage, or better transcripts in noisy audio.
- You are evaluating how to move a speech model from training into Riva or NIM across cloud, on-premises, edge, or embedded infrastructure.

## Related talks

- [Why TTS Models Now Look Like LLMs](https://aietalks.com/talks/why-tts-models-now-look-like-llms) (Samuel Humeau, Mistral, 22:26)
- [Optimizing Inference for Voice Models in Production](https://aietalks.com/talks/optimizing-inference-for-voice-models-in-production) (Philip Kiely, Baseten, 15:13)
- [Serving Voice AI at Scale](https://aietalks.com/talks/serving-voice-ai-at-scale) (Arjun Desai, Cartesia & Rohit Talluri, AWS, 17:05)
- [Engineering voice agents: Latency, quality, and scale](https://aietalks.com/talks/engineering-voice-agents-latency-quality-and-scale) (Rishabh Bhargava, Together AI, 24:35)
- [From Text to Vision to Voice: Exploring Multimodality with OpenAI](https://aietalks.com/talks/from-text-to-vision-to-voice-exploring-multimodality-with-openai) (Romain Huet, OpenAI, 23:39)