# Stop Renting Your Cognitive Infrastructure Thiyagarajan Maruthavanan, Kalmantic Labs | AI Engineer World's Fair 2026 | 07:52 Source: https://www.youtube.com/watch?v=Bck7ABCZRZI Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/stop-renting-your-cognitive-infrastructure Published: 2026-07-18 Tags: cost, deployment, enterprise, inference ## TL;DR - Inference costs can grow beyond a team's mental budget because rented platforms make usage feel like prepaid credits rather than a fixed monthly bill. - Renting is reasonable before product-market fit, but post-product-market-fit products and enterprise projects need owned inference infrastructure for control, audits, and reproducibility. - Thiyagarajan Maruthavanan's rule is to rent while learning and own the infrastructure when the business depends on the workload. ## Summary Thiyagarajan Maruthavanan argues that teams lose control when they treat inference as a cheap, metered service. He describes his own costs rising after an app spread widely, then recounts a stolen API key that drove spending from $7,000 to $10,000 before his co-founder stopped it. Moving applications and agents onto a DGX box reduced dependence on Anthropic, but exposed memory and reliability limits. Three enterprise customers showed why rented token factories can fail: an investment fund needed control over rate limits, a hospital could not pass an audit with a third-party dependency, and a tax practice needed access to model internals to reproduce recommendations. His advice is to rent models while testing demand, then build or own inference infrastructure after product-market fit or when an enterprise project has already been budgeted. He also introduces Just Token Max, an open-source project for reducing input-token and context costs. ## Key ideas ### Rented inference makes overspending easy to miss [00:00](https://www.youtube.com/watch?v=Bck7ABCZRZI&t=0s) Thiyagarajan compares rented intelligence platforms with prepaid casino credits. Users load money, start using the system, and can pass the spending limit they had in mind before noticing. He saw this with an app that generated prompts for songs from Suno.com. After sharing it, the app reached hundreds of thousands of users and inference costs rose into the hundreds of thousands of dollars. He also describes an API key being stolen and used from China. Spending rose from $7,000 to $7,500 and then $8,000, stopping at $10,000 only after his co-founder intervened. ### Agent workloads expose weaknesses in generic endpoints [01:50](https://www.youtube.com/watch?v=Bck7ABCZRZI&t=110s) The cost problem is not only the price of a model call. Thiyagarajan points to uncompressed input tokens, context that is not managed well, and agent loops that make many wasteful calls. A rented endpoint does not understand the shape of the workload, so it cannot automatically account for these patterns. This makes an application with repeated agent calls especially vulnerable to costs that grow faster than expected. His experience with a widely used music-related app gave him a direct reason to examine inference infrastructure rather than treating the endpoint as a fixed utility. ### A local token factory can reduce dependence but still has limits [02:28](https://www.youtube.com/watch?v=Bck7ABCZRZI&t=148s) Thiyagarajan describes the common alternative to Anthropic and OpenAI as a token factory. Teams can use open-source models on cloud infrastructure, buy inference from a provider, or assemble GPUs locally. He bought a DGX box and moved Ulta Sono from Anthropic onto it. The setup worked well enough for new applications and a growing set of research-lab agents, but memory became a bottleneck. When three enterprises asked him to reproduce the setup, reliability became a larger concern. A local box that works for a lab does not automatically meet enterprise requirements. ### Enterprise customers need control beyond a lower bill [03:38](https://www.youtube.com/watch?v=Bck7ABCZRZI&t=218s) Three potential enterprise deployments exposed different reasons to own infrastructure. An investment fund did not want another party deciding how many requests its email-based investment analyst could make, so control over rate limits mattered. A hospital found that a third-party vendor dependency was rejected during an audit. A tax practice needed to recreate an intelligent recommendation, but could not do that without access to the model's internals. These cases show that infrastructure decisions can affect governance, audit approval, and reproducibility, even when the application itself works. ### The point to stop renting depends on product maturity [04:53](https://www.youtube.com/watch?v=Bck7ABCZRZI&t=293s) Thiyagarajan says a startup that is still testing whether a use case has demand can rent infrastructure. Once the product has found product-market fit, he says the team cannot afford to ignore building its own infrastructure. The same applies to an enterprise that has already budgeted a project, because that budget assumes the use case is viable. He explains the transition with housing: someone entering a new city may rent or use Airbnb while learning the area, but cannot raise a family in an Airbnb. Inference ownership follows the same progression from experimentation to dependence. ### Inference costs can be optimized at several layers [05:45](https://www.youtube.com/watch?v=Bck7ABCZRZI&t=345s) After deciding to build infrastructure for his applications, agents, and future scaling, Thiyagarajan found optimization opportunities even before hardware ownership. He mentions input-cost and token management, along with context management. He combined some of these experiences in Just Token Max, an open-source project he presents as an alternative to Netflix's Headroom. He says his team benchmarked Just Token Max against Headroom and found it superior on many parameters. He also wrote the book Peak Inference: Infraeconomics of AI Inference for people considering their own inference infrastructure. ### Different vendors promote the layer they sell [06:42](https://www.youtube.com/watch?v=Bck7ABCZRZI&t=402s) Thiyagarajan says the AI market changes every three to six months, which makes its advice noisy. He contrasts Jensen Huang's token-factory view with Satya Nadella's idea of unmetered, local intelligence, then describes new cloud and inference-endpoint providers arguing that their layer will capture the value. His own experience across applications, agents, scaling, and infrastructure leads to a simpler rule: rent while learning and own when earning. He presents that as a decision framework, while still telling the audience to reach its own answer. ## Notable quotes - "It's almost as if you're loading credits inside a casino." (00:35) - "You cannot raise a family in an Airbnb." (05:28) - "If you want to learn, you can rent, but if you want to earn, then you have to own." (07:13) - "Rent to learn, own to earn." (07:18) ## Tools & references mentioned - Anthropic - OpenAI - Suno.com - Ulta Sono - DGX - Just Token Max - Headroom - Netflix - Jensen Huang - Satya Nadella - Peak Inference: Infraeconomics of AI Inference - Kalmantic Labs ## Who should watch - You are running an AI product whose inference bill grows with user activity and you need to decide whether to keep paying an API provider. - Your team is moving from experiments to a product with established demand, or an enterprise project has already received a budget. - You need control over rate limits, audit requirements, model reproducibility, or the infrastructure behind agent workloads.