Token-based text-to-speech models predict audio tokens from text and previous audio, using codebooks to represent sound in discrete form.
2
A single-speaker YouTube video can be transcribed with Whisper, corrected, split into clips of up to 30 seconds, and used to fine-tune Sesame's CSM-1B model.
3
Combining LoRA fine-tuning with voice cloning produced a voice that sounded more like Ronan McGovern, although the small dataset left problems with pacing, accent, and intonation.
Summary
Ronan McGovern walks through a Colab notebook for adapting Sesame's CSM-1B text-to-speech model to a voice taken from a YouTube video. He first explains how token-based speech models turn audio into discrete codebook entries, with Sesame predicting one token through its main transformer and 31 more through a depth decoder. The practical workflow downloads a single-speaker video, transcribes it with Whisper, corrects errors, and combines segments into clips of up to 30 seconds. The notebook then applies LoRA adapters through Unsloth, formats the audio and text fields, and trains for one epoch on a small dataset. McGovern compares zero-shot generation, voice cloning, and fine-tuning. The fine-tuned model produced a recognisably male, somewhat Irish-sounding voice, while cloning combined with fine-tuning gave the closest result. He is clear about the limits of the basic setup and recommends cleaner sentence boundaries, evaluation data, monitoring, and more training examples.
Token-based speech models predict audio tokens from text and prior audio
McGovern explains text-to-speech by comparing it with models such as GPT-4o and Llama. A text-to-speech model takes text tokens, along with earlier text and audio, and recursively predicts the next audio token. Those tokens are decoded into speech. The model therefore needs to accept text and audio while producing a sequence of audio tokens. This differs from a language model that predicts the next text token, although the autoregressive idea is similar.
Audio codebooks turn sound waves into discrete representations
A codebook works like a dictionary whose entries are vectors for pieces of sound. An encoder converts a waveform into tokens, and a decoder turns those tokens back into a waveform. During training, the reconstructed sound is compared with the original, and the difference is backpropagated through the model. McGovern says one token per time window is not detailed enough, so token-based audio systems use multiple tokens at each window, often arranged hierarchically to capture both broad acoustic information and finer detail.
Sesame CSM-1B uses a main transformer and a depth decoder
The Sesame model uses 32 tokens for each audio window. Its main transformer predicts the zeroth token autoregressively. A second, much smaller transformer predicts the other 31 hierarchical tokens. McGovern describes the system as having a main one-billion-parameter model plus the smaller decoder, along with components that convert waveforms to tokens and tokens back to waveforms. This architecture is why the notebook loads more than a single ordinary language-model transformer.
A small single-speaker dataset can change the model's voice
The notebook downloads a YouTube video, uses Whisper to transcribe it, and creates audio-text examples. The examples contain clips of up to 30 seconds. McGovern has 41 rows in his run and says that roughly 50 clips of this length can start to affect quality. He recommends choosing a video with one speaker because the basic notebook does not support diarization. He also says that 10-second clips and only 20 examples did not provide enough data in his earlier experiments.
Transcript cleanup and sentence boundaries affect the training examples
Whisper's JSON output should be read and corrected before it becomes training data. McGovern gives the example of correcting the spelling of the Trellis Research name, since the ordinary word and the company name use different spellings. The notebook then joins Whisper segments until they reach about 30 seconds. He points out that this simple method can cut rows at awkward places. Ending each row at a full stop, using NLTK or a regular expression, would produce more complete text units.
Voice cloning and fine-tuning solve different parts of the problem
Zero-shot inference can produce a different speaker each time because the sampling temperature is nonzero. Voice cloning instead passes an audio sample to the model, which guides the generated voice toward that sample. McGovern's cloned output is closer to his voice than the zero-shot examples, but he expects fine-tuning to improve it further. He later combines both methods, using the fine-tuned model with a voice sample, and gets the closest result of the three approaches.
McGovern applies LoRA adapters to the attention and linear layers, including Q, K, V, and MLP projections. He uses alpha 16 and rank 32, with rescaled LoRA learning rates. The resulting setup trains just under 2 percent of the model's parameters. He does not train the embeddings because the token vocabulary is not changing. CSM-1B is only a few gigabytes, so he says quantization is unnecessary on a T4 with 15 GB of memory.
The workshop run improves the voice but still needs better evaluation and data
The example trains for one epoch on 41 rows, with a virtual batch size of eight. McGovern reports that training loss falls from about 6.34 to about 3.7, then reaches 3.72 in the completed run. The resulting voice sounds recognisably like him and has an Irish quality, but its pacing and some pronunciation remain inconsistent. He recommends splitting off evaluation data, checking evaluation loss, monitoring gradient norm with TensorBoard, filtering pauses, and increasing the dataset toward 500 rows of 30-second clips.
"But you can see here that if you combine fine-tuning with cloning, you're able to get pretty good performance even with a relatively small amount of data."33:04
Who should watch
You want to adapt a token-based text-to-speech model to a voice from an existing YouTube video and need a runnable Colab workflow.
You are comparing voice cloning with parameter-efficient fine-tuning and want to hear how the two methods differ.
You need practical advice on transcript cleanup, clip construction, LoRA settings, and basic checks for a small speech dataset.
