AI inference is improving quickly enough to change how software is built and used.
2
Very fast models could make personalized, multimodal agents, background analysis, and complex decision-making practical.
3
Groq expects cheap, low-latency AI to affect education, security, enterprise software, and decentralized computing.
Summary
Sunny Madra uses Intel's 1999 1-GHz milestone as a comparison for the changes coming to AI inference. Faster models already let systems process large inputs and complete research-heavy tasks in seconds. If latency falls much further, Madra argues that an LLM could become the core of computing, changing how software is built, run, and scaled. He describes possible applications including multimodal interfaces, advanced virtual assistants, agents that work together, background analytics, personalized content, automated data science, supply-chain optimization, edge and decentralized AI, and protective systems against voice scams. He connects this future to Groq's goal of making tokens cheap and widely available. The talk is speculative in places, but it includes concrete examples from Globe Engineer, Definitive Intelligence's Pioneer, and personalized education services. Madra's central point is that inference speed affects what kinds of software are practical, rather than only making existing chat interfaces respond faster.
AI inference is improving at a pace that could change software's basic assumptions
Madra begins with Intel's 1999 1-GHz processor milestone. He says microprocessors gained three orders of magnitude in speed over about two decades, while current LLM innovation may be moving faster than Moore's Law. At Groq, the team increased the speed of Llama 3 8B by over 50% between April and June 2024. He presents this as the start of a period where inference speed could change what software does, rather than simply making existing programs run faster.
Fast models already compress hours of research into seconds
Madra describes Globe Engineer, a service that takes a task such as planning a trip to New York and researches flights, taxis, hotels, food, and an itinerary while connected to the internet. He says the result can arrive in less than five seconds. Without such a system, he might open tens or hundreds of browser tabs, with each tab representing a separate research stream. Faster input and output token processing turns that work into a simple interface and lets an LLM integrate information faster than a person can.
An LLM could become the core layer through which computing is used
Madra compares the possible shift to the Industrial Revolution. Earlier computing digitized paper processes, the internet connected them, and cloud and mobile computing changed their form factor. He says AI is beginning to industrialize technology itself. A designer who once made one or two presentation images a day can now use Midjourney to generate a thousand in a minute. At lower latency, the LLM could sit at the center of video, audio, browsers, code interpreters, and file systems.
Near-instant inference would make personalized and multimodal software practical
Madra says current responses are often close to reading speed, while instantaneous responses would change what agents can do. Personalization could include details about a person's pets, children, or spouse, with speed making that interaction feel continuous. He also expects natural language interfaces to accept any kind of input and return any kind of output. For example, someone could ask by voice about haircut times and receive the available times as text, which is easier to remember than spoken options.
More inference speed supports agents that work together and run continuously
Madra argues that LLMs are often measured with single-shot tasks because performance and cost make longer interactions difficult. He says existing models can score much better when given multiple attempts, and that papers have shown multiple agents can let a smaller model compete with a larger one through repeated reasoning and collaboration. He expects more complex task scheduling, collaborative agents across devices, and agents that continuously watch incoming data once compute cycles become inexpensive.
Continuous agents could automate analysis and discover patterns people did not ask for
Using Definitive Intelligence's Pioneer, Madra describes an automated data science agent designed to run almost endlessly on a business problem. A company can define a KPI, supply its incoming data, and let the agent ask follow-up questions, analyze changes, and produce findings instead of waiting for people to make spreadsheets and presentations. On a dataset of employee performance reviews, Pioneer found a relationship involving age, review type, and productivity that the team had not thought to investigate.
Low-latency AI creates uses in distributed computing and personal security
Madra mentions Hyperspace, which lets people make unused GPU capacity available for cloud workloads that do not need real-time responses. He expects more distributed AI as systems gain throughput and lower latency. He also describes AI-assisted voice scams that use detailed personal narratives and convincing cloned voices. A protective system running quickly on a person's side could help verify callers. In his example, asking the caller to send a formal message through the HSBC app exposed the scam.
Cheap personalized inference could change education and enterprise integration
Madra connects Groq's aim of making tokens cheaper and more available to personalized tutoring. Citing Sal Khan's two-sigma discussion, he says personalized tutors can improve students' test performance by two standard deviations. He gives the example of teaching arithmetic through a child's interests, such as ponies and unicorns. He also says fast, inexpensive AI could reduce the enterprise burden around interconnectivity, interoperability, and compatibility, which account for much of the cost of deploying and maintaining enterprise software.
"If we can get into like 10,000 complex decisions per second just by getting this down to 0.1 milliseconds, then it does become viable to think about the core of our computing becoming an LLM."06:47
Who should watch
You are building agents and need to understand how latency and token cost affect the tasks they can handle.
Your product depends on multimodal interaction, personalization, continuous background analysis, or multiple agents working together.
You work on inference infrastructure, edge AI, enterprise software, security, or personalized education and want examples of possible applications.
