An AI assistant can use personal context to turn a voice request into a phone conversation with another person.
2
Convex reactive queries let the client display request status and call transcripts as the server writes updates to the database.
3
Phone-call latency comes from several services in sequence, so the prototype uses pre-recorded audio, Opus encoding, prompt reduction, and faster speech services to improve response time.
Summary
Tom Redmond builds Floyd, a proof-of-concept personal assistant that receives voice requests, understands them with GPT-4, calls a business through Twilio, and speaks with the person who answers. The assistant stores user context, such as a child's school or a preferred mechanic, then saves a snapshot of the relevant context with each request. Google Cloud transcribes audio, OpenAI manages the conversation and generates speech, and Convex stores the request, status changes, context, and transcript. Redmond demonstrates calls to a school and a flower shop, including a case where Floyd stops when it lacks delivery details and reports back by text. Much of the workshop explains how an Express server and a Convex client coordinate through reactive database queries. Redmond is clear that the code is a prototype. Latency remains the main problem, especially as conversation prompts grow and text-to-speech responses take several seconds.
A useful personal assistant needs to handle conversations with people
Redmond starts from the limits of current assistants, which mostly set calendar events, reminders, and timers. He argues that a personal assistant should know facts about the user and handle ordinary phone and email work. Floyd can store details such as the user's car, mechanic, or children's school, then use those details when it needs to call someone. The assistant is meant to ask for missing information rather than invent it. Redmond also wants the system to be honest with the person on the other end of the call that it is an AI.
Floyd turns a voice request into a reactive database workflow
The client streams the user's speech to Google Cloud for real-time transcription, then saves the user and request in Convex. The server subscribes to new requests with a reactive query, so it does not poll for work. It changes the request status to 'in progress' and writes status updates back to the database. The client is subscribed to matching requests and receives those changes automatically. Redmond describes this as a handoff between the client and server through database state, with the client able to show progress as the request is processed.
The assistant snapshots relevant user context before asking GPT-4 to act
Before making a call, the server looks up what it knows about the requester. It stores a moment-in-time context snapshot on the request, then sends the request and the relevant context to GPT-4. In the school example, GPT-4 extracts Mara Redmond's name, the school, the reason for the absence, the date, and the steps needed to complete the task. That action plan becomes additional context for the phone conversation. If the phone number is not already in the user's history, the intended design is to search for it or text the user for a preferred vendor.
The phone conversation is a repeated audio transcription and speech loop
Twilio makes the call and streams audio from the other person to the server. Google Cloud transcribes that audio, and the transcription goes back to GPT-4 so it can decide what to say next. OpenAI text-to-speech turns the response into audio, which the server streams back through Twilio. This loop repeats until the conversation is complete. Each transcript segment is appended to Convex while the call runs, allowing the client to display the conversation in real time. Redmond says the prototype works, but its latency still needs substantial improvement before production use.
The demos show both successful calls and a controlled escape when information is missing
In the first demo, Floyd calls Mara's school and explains that she is staying home sick. It answers a follow-up question about when she may return and says Tom will provide updates. In the second demo, Floyd calls a flower shop to order flowers for Christa. When the shop asks for delivery details and the order reaches $560, Floyd cannot complete the purchase because it lacks the needed information. It ends the call and is designed to text Tom with the missing details and the price. Redmond presents this stop-and-report behavior as safer than guessing.
Redmond shows that the request dashboard uses a single reactive query. When a request changes, the React client receives the new data automatically. The same mechanism streams transcript entries into the interface as the server appends them. He contrasts this with earlier attempts to stream the phone call directly to the browser, which became difficult with Next.js, Vercel, Socket.IO, and WebSockets. Moving the state the client needs into Convex lets an Express server handle the Twilio connection while the client listens for database changes.
The prototype can put backend functions and frontend code in one TypeScript codebase
Convex functions can define queries, mutations, HTTP actions, schemas, and database access alongside frontend code. The generated API is type-safe, and a call such as 'api.requests.get' targets a function deployed to Convex rather than shipping that function to the browser. Redmond explains that the function runs on a Convex server next to the database, using a custom V8 engine. He gives developer ergonomics as the main reason to choose Convex over a conventional database such as Postgres. The prototype also uses an Express server because it needs a place to manage Twilio media streams.
Latency grows when every turn sends more conversation to the model
Redmond measures the time spent transcribing audio, getting a response from OpenAI, and converting that response to speech. He observes that prompts often get slower as the conversation grows because they include more previous turns. He had built the system before OpenAI's Threads API was available and says he would work to reduce the amount of prompt text sent each time. Long text-to-speech inputs can also take two to five seconds. Other possible changes include paying for lower-latency speech services, testing different models, and using newer multimodal systems that accept audio directly.
"I realized I feel like there's enough technology out there these days that we could actually string together a number of platforms such that you could have an AI assistant that knows about you."01:31
Who should watch
You are building a voice agent that must move between a browser, speech APIs, an LLM, and a phone network.
You want a concrete example of using a reactive database to pass work and live status between a frontend and a server.
You need to understand where latency appears in a phone conversation and what a prototype can do when it lacks information.
