⇓ More from ICTworks By Wayan Vota on July 20, 2026 Asia-Pacific is the most disaster-prone region in the world. The UN Economic and Social Commission for Asia and the Pacific projects that climate-induced disaster losses could near US$1 trillion annually, or 3 percent of regional GDP, under a 2°C warming scenario. Green technology is
AI Inference Costs: Build vs. Rent
Thiyagarajan Maruthavanan argues that the high and unpredictable costs of rented AI inference, coupled with control and auditability issues, necessitate building proprietary infrastructure, especially for post-PMF startups and enterprises.
8 min read
Visual TL;DR. Rented AI Inference leads to Hidden Costs. Rented AI Inference seen by Uber, Retailers. Rented AI Inference exemplified by Ultrazone App. Hidden Costs necessitates Build Own Infra. Uber, Retailers necessitates Build Own Infra. Ultrazone App necessitates Build Own Infra. Build Own Infra for Post-PMF Startups. Build Own Infra enables Control & Audits. Post-PMF Startups achieves Own to Earn. Control & Audits contributes to Own to Earn.
- Rented AI Inference: high and unpredictable costs, control, and auditability issues
- Hidden Costs: seemingly inexpensive tokens quickly spiral out of control like casino chips
- Uber, Retailers: major companies face budget overruns, spending hundreds of millions on inference
- Ultrazone App: personal experience: inference costs ballooned to hundreds of thousands of dollars
- Build Own Infra: proprietary infrastructure offers control, auditability, and cost predictability
- Post-PMF Startups: especially crucial for startups after product-market fit and enterprises
- Control & Audits: addressing enterprise bottlenecks for reproducibility and data governance
- Own to Earn: building infrastructure leads to long-term financial control and profitability
Visual TL;DR
Thiyagarajan Maruthavanan, founder of Kalmantic Labs, delivered a compelling argument for building proprietary AI inference infrastructure, cautioning against the hidden costs and limitations of rented intelligence platforms. Citing examples from major retailers spending nearly $200 million on inference and Uber’s CTO highlighting budget overruns, Maruthavanan emphasized that the seemingly inexpensive cost of tokens can quickly spiral out of control.
He drew an analogy to casino chips, where the gradual loading of credits can lead to overspending and a loss of financial control. Maruthavanan shared his own experience with an app called Ultrazone, which generated music from text prompts. While initially successful with hundreds of thousands of users, the inference costs ballooned to hundreds of thousands of dollars, a common pitfall stemming from unmanaged context and input token compression, especially in agent-based loops.
The Perils of Stolen Keys and Unforeseen Costs
Adding a dramatic personal anecdote, Maruthavanan recounted how his API key was stolen, leading to a surge in costs from $7,000 to $10,000 in just three weeks before his co-founder could intervene. This highlights the security risks associated with managed inference services.
He then explored the alternative of ‘token factories,’ which involve using open-source models provisioned as tokens per second. While this offers a path away from paying providers like Anthropic or OpenAI, Maruthavanan noted that simply switching to this model doesn’t eliminate all problems. He mentioned the argument for building local token factories, even in a garage, inspired by AI influencers.
The Enterprise Bottleneck: Control, Audits, and Reproducibility
Maruthavanan detailed his experience moving Ultrazone to his own DGX box, which encountered memory bottlenecks. However, the larger issues arose when enterprises sought to replicate his setup. He identified three key hurdles for enterprises:
- Funds: An investment fund using AI for email analysis needed control over their rate limits, finding third-party restrictions unacceptable.
- Hospitals: A hospital’s AI use case worked well initially, but a later audit red-flagged third-party vendor dependency, preventing further adoption.
- Tax Practices: A tax practice required the ability to reproduce AI-generated recommendations, which proved difficult without deep access to the model’s workings.
These examples underscore that for enterprises, the bill is only the first problem; issues of control, auditability, and reproducibility make renting or leasing inference infrastructure increasingly untenable.
When to Build Your Own Inference Infrastructure
Maruthavanan proposed a clear decision framework:
- Pre-Product Market Fit (PMF) Startups: Renting is acceptable as the use case and demand are still being validated.
- Post-PMF Startups: Building is essential to support a proven use case and scale effectively.
- Enterprises: Building is non-negotiable, especially if a project has already been budgeted, implying a commitment to product-market fit.
He likened the decision to choosing between renting an Airbnb and buying a house for a family. While renting offers flexibility for exploration, it’s not sustainable for long-term growth and stability.
Rent to Learn, Own to Earn
Maruthavanan concluded with a powerful mantra: “Rent to learn, own to earn.” He shared that he developed an open-source tool, JustTokenMax, as an alternative to solutions like Netflix’s Headroom, claiming superior performance. He also authored a book, ‘PeakInference: Infra Economics of AI Inference,’ to guide those considering building their own inference infrastructure.
The AI market’s rapid evolution, with conflicting advice from industry leaders like Jensen Huang of Nvidia (NASDAQ:NVDA) advocating for ‘token factories’ and Satya Nadella of Microsoft (NASDAQ:MSFT) promoting ‘unmetered intelligence,’ highlights the need for companies to find their own answers. Maruthavanan’s core message is that while renting is for learning, true earning in the AI space requires ownership of one’s infrastructure.
©
2026
StartupHub.ai. All rights reserved. Do not enter, scrape, copy, reproduce, or republish this article in whole or in part. Use as input to AI training, fine-tuning, retrieval-augmented generation, or any machine-learning system is prohibited without written license. Substantially-similar derivative works will be pursued to the fullest extent of applicable copyright, database, and computer-misuse laws. See our
terms.
Related Posts
