Gemini can combine PDFs, images, video, audio, URLs, and search results in one analysis request, then produce a cross-modal summary.
2
An agent can use Gemini as a reasoning model that decides when to call image and speech generation tools instead of following a fixed NotebookLM-style workflow.
3
Native image, speech, and live audio models use Gemini's understanding of the world, language, accents, and tone to create more context-aware outputs.
Summary
Patrick Löber explains how to build an any-to-any multimodal agent with Gemini. The system first gathers and analyzes sources such as PDFs, videos, audio files, images, URLs, and search results. Gemini can connect information across those sources and produce a study guide or summary. A reasoning loop then decides which parts need extra material, calling image-generation and speech-generation tools through function calling. Löber uses a NotebookLM-style application as the example, with generated infographics and podcast-style audio. He also explains practical details such as file uploads, timestamp ranges, audio transcription, large context limits, and context caching for repeated queries. The final component is the Live API, where a native audio-to-audio model handles real-time interaction without a cascaded speech pipeline. The talk is a practical architecture walkthrough, although Löber is clear that the Gemini ecosystem still consists of several specialized models rather than one universal model.
Gemini's any-to-any vision still uses several specialized models
Löber defines any-to-any as taking in many forms of input and producing many forms of output. Gemini can understand text, code, images, audio, video, URLs, and Google Search results. It can generate text, images, speech, video, function calls, and code. He qualifies the diagram by saying this is not one multimodal model yet. The main Gemini model handles multiple input modalities but outputs text, while specialized models handle native image and speech generation. Gemma also allows developers to build multimodal agents locally, with capabilities depending on the model size.
A NotebookLM clone works better as a reasoning agent than a fixed workflow
The example application accepts multiple sources and produces a study experience with summaries, podcast-style audio, and infographics. Löber chooses an agentic design instead of hardcoding every step. Gemini acts as the reasoning model and decides what material to create. It calls specialized models through tools, then checks whether the generated assets are sufficient or whether it needs more. The application can combine information from PDFs, images, videos such as lectures, and voice memos, allowing the model to connect information across source types.
Multimodal understanding can use common file and inline-data inputs
The implementation uses the Google AI SDK and a client models generate content call. Developers can upload PDFs, video files, and MP3 files, or pass smaller files as inline data. The same request can place all the sources in a contents list and ask Gemini to analyze them together. Löber also mentions Gemini API Skill, which lets an agent use the Gemini models without the developer memorizing the SDK code. The model can summarize the sources or transcribe an audio file when instructed in the prompt.
Long audio and video inputs need practical controls
Löber gives the context calculations behind the file workflow. One minute of audio uses about 1,920 tokens, and a one-million-token limit therefore allows more than nine hours of audio. Video allows roughly one hour under the stated setup. Developers can adjust configurations, analyze only a timestamp range such as minutes five through fifteen, and upload larger files through the File API. YouTube URLs can also be passed directly. Context caching is useful when a long file is queried repeatedly, because Löber says it saves 90% of the costs.
The generation phase lets the agent choose which assets to create
After Gemini builds an understanding of the source material, a second phase uses an agentic loop for generation. Gemini remains the reasoning model and calls native generation models through function calls. It can decide whether a concept needs an infographic or whether a section would benefit from an audio explanation. Löber shows that the image model can create infographics from a prompt. The speech model can produce a two-speaker audio file, which fits the podcast format used by the NotebookLM-style example.
Function declarations give the agent controlled generation tools
The tool setup starts with function declarations. Each declaration has a name, a description, and parameters. Löber's image function takes one string containing the detailed image prompt, and the audio function is configured in the same way. The agent prompt includes the synthesized study material and tells Gemini to act as a research partner. It asks the model to decide which complex concepts need visual diagrams and which sections need audio summaries, then call generate image or generate speech as appropriate.
Native generation models use Gemini's world and language understanding
Löber says native image models are based on Gemini, so they benefit from training that also goes into the main Gemini models. His map example has arrows pointing toward a location, and the model generates the Golden Gate Bridge because it understands what the arrows indicate rather than matching pixels. He also shows an educational example where an image model corrects math homework and adds visual corrections. The audio models understand multiple languages, accents, and tone, with examples in British and Bavarian accents.
The Live API provides real-time interaction through a model Löber describes as audio to audio. Audio goes in and audio comes out through one architecture, instead of passing through a cascaded pipeline of separate models. He says this supports natural-sounding interaction and points viewers to AI Studio Live for experimentation. The example includes a user asking whether Gemini can see them, followed by a spoken description of the user's hair, beard, jacket, and shirt. Löber presents the pattern as transferable to other application areas.
