Todd Fisher built a JUCE plugin that turns guitar notes into spoken words by combining speech synthesis with guitar-triggered playback.
2
Automatic word segmentation is difficult because spoken words often have no silence between them, so his system combines energy gaps with vowel-based syllable detection and still needs manual edits.
3
A microphone, speech recognition, local language model, and preprocessed vocal samples let the guitar answer questions and move toward singing, although the singing pipeline is too heavy to run live.
Summary
Todd Fisher describes a long-running project to make a guitar speak. He places it in the history of guitar technology, from pickups and amplifiers to effects pedals, Peter Frampton's talk box, and software plugins. His implementation is a JUCE audio plugin for a DAW. The first version played a text-to-speech clip whenever he played a note. He then worked on cutting speech into individual words, where simple silence detection failed because speech often runs continuously. A combination of energy-gap segmentation and sonority-peak syllabification worked reasonably well, but he still edited boundaries by hand. For singing, he detects each guitar note with the YIN pitch algorithm, synthesizes a tone, and runs it through a vocoder. A later version uses speech recognition and a local model to answer audience questions through the guitar. Pitch-shifted samples from VocalSet, processed with WORLD, get closer to singing, but the processing must be prepared ahead of time.
Live performances gave Fisher a reason to build theatrical instruments
Fisher opens with performances that stayed in his memory: a Slipknot drummer playing upside down while his drum kit tipped onto a wall, and a Stranger Things Broadway scene that created a convincing slow-motion effect in real time. He connects those experiences to engineers' long lists of projects they hope to build. In his view, recent AI tools make it easier to move from an idea to a working experiment. His aim is to encourage people to start a personal project because it can teach them something or help solve a problem they care about.
The guitar's history points toward software and AI as its next modification
Fisher traces a sequence of changes to the guitar. A pickup connected the instrument to an amplifier and made rock music louder. Effects pedals then let players combine and reshape sounds. Peter Frampton's talk box sent guitar audio through a physical tube into his mouth, allowing him to form words with the sound. Software later moved many effects into tools such as Pro Tools, Logic, and Fruity Loops. Fisher asks what a guitar's next evolution might be when AI is added to this chain.
A Halloween lighting project became the starting point for a speaking guitar
While playing guitar for trick-or-treaters, Fisher dressed as Eddie Munson from Stranger Things and built an app that projected the show's alphabet onto his garage door. Guitar notes communicated with the lights, and he could set custom messages such as 'Happy Halloween.' The project was playful, but it led him to ask how hard it would be to make his guitar speak. That question became the longer audio project he presents in the talk.
The first JUCE plugin made one guitar note trigger a spoken clip
Fisher built the instrument as a JUCE plugin that could be inserted into Logic like any other audio effect. He used text-to-speech tools including Piper and Apple's built-in speech capabilities. The initial pipeline accepted raw text, generated an audio clip, and played that clip when he played a guitar note. The one-word demo says 'Developers' repeatedly. The basic trigger worked, but it exposed the next problem: normal speech contains multiple words, so the audio had to be divided into smaller pieces.
Word segmentation fails when speech has no silence between words
Fisher first tried energy-gap segmentation. The method looks for points where a waveform approaches silence and cuts there. That works when words have clear gaps, but running speech often has no silence between adjacent words. He then investigated sonority-peak syllabification, which looks for syllable peaks associated with vowels. Combining the syllable method with energy gaps produced usable results, though the boundaries were still imperfect. He finished the process by letting himself drag segment boundaries manually.
YIN pitch detection lets guitar notes drive a synthesized voice
To move from speaking toward singing, Fisher needed to identify the fundamental frequency of each fretted guitar note. He uses the YIN pitch algorithm for that detection. The detected pitch controls a synthesized note, whose sound is shaped with an ADSR envelope. He then pushes the synthesized tone through a voice clip and a vocoder, following the basic idea of a talk box. The result mixes the synthesized note with the AI voice, with a clarity control used to balance the two signals.
A microphone and local model let the guitar answer an audience question
Fisher extended the plugin into a conversational system. He speaks into a microphone, converts the speech to text with Whisper, sends the text to a local language model, and routes the model's response back through the guitar. When someone asks, 'What is reality?', the guitar gives an existential reply about music and understanding reality. The result is choppy, and Fisher is direct that it is not singing yet, but the complete loop works: spoken question, local model response, and guitar output.
Preprocessed vocal samples sound closer to singing than live synthesis
For a more convincing singing voice, Fisher uses recordings from the open VocalSet dataset. He processes the samples with WORLD to shift their pitch, then maps the prepared sounds to notes on the guitar. The processing is too heavy to perform live, so the samples have to be pre-baked. His demonstration starts with the five vowel sounds rather than complete lyrics. The output is strange and still far from an opera singer, but it is closer to singing than the earlier spoken-word system.
"The issue with that is there's actually sometimes when I'm speaking right now, for example, there's actually no silence in between some of my words."08:25
Who should watch
You want to build an audio or music experiment and need a concrete example of taking it from a simple trigger to a more complex pipeline.
You are working with speech segmentation and need to see why silence-based word boundaries break down in continuous speech.
You want to connect speech recognition and a local language model to a physical instrument, while keeping the limits of live audio processing in view.
