The Inference Paradox

The cost of running an agentic AI workflow will rise more than fivefold through 2028.

In the same breath, Gartner expects the per-token price of frontier inference to fall by more than 90 percent by 2030.

The cost of the thing you actually buy, a completed workflow, goes up.Will Sommer, the Gartner analyst behind the forecast, said it plainly: routing a task to an agentic reasoning model increases inference costs at least fivefold, and…

It is what you get when four things compound upward and one thing compounds downward, and nobody in the building owns the ratio. 02 · The paradox in one equation ## Four terms up, one term down Gartner built its forecast on a Tokenomics…

Sometimes two. Jensen Huang said on NVIDIA's fourth-quarter fiscal 2025 call that reasoning can demand 100 times more compute than a simple query, that the vast majority of NVIDIA's compute today is actually inference, and that future…

Benchmarking GPT-4o cost $109. o1-pro and GPT-4.5 were priced at $600 per million output tokens against $15 for Claude 3.5 Sonnet.

On a medium-complexity coding prompt, o1 ran at $0.450 per request and o3 at $0.364 against $0.011 for GPT-4, roughly 33 to 40 times more.

Even a cheap reasoning model loses its headline advantage once the reasoning tokens are counted: a single DeepSeek R1 request can generate 5 to 50 times the tokens of V3.1 (DeployBase). 02 ### Agents burn more tokens than chat.

Anthropic also found that token usage by itself explains 80 percent of the variance in performance on the BrowseComp evaluation.

Runs on the same task differed by up to 30 times in total tokens, and higher token usage did not translate into higher accuracy (Microsoft Research).At the workflow level the same thing shows up in ordinary examples.

Seventeen times more for what the user experiences as one question answered.

At 30 turns the actual cost is 15.5 times the naive linear estimate (n1n.ai). 03 ### Deflation is real, fast, and resets at every new tier. The deflation half of the paradox is well documented and in places runs faster than Gartner says.…

The same piece notes that OpenAI's o1 launched at the same $60 per million output tokens that GPT-3 charged at launch (a16z, LLMflation).Epoch AI tracked the price of reaching a fixed capability across six benchmarks and found declines…

The same piece notes that OpenAI's o1 launched at the same $60 per million output tokens that GPT-3 charged at launch (a16z, LLMflation).Epoch AI tracked the price of reaching a fixed capability across six benchmarks and found declines…

Epoch's read of the trend is very roughly a 5 to 10 times cost reduction per year for reaching a given capability level (Epoch AI, Feb 2026).So the price of yesterday's capability collapses.

OpenAI wrote it down in the o1 announcement: the performance of o1 consistently improves with more reinforcement learning and with more time spent thinking.

Snell and colleagues at UC Berkeley and Google DeepMind formalized it as a scaling law: scaling test-time compute can be more effective than scaling model parameters (Snell et al., 2024).When the law says more inference compute buys…

SemiAnalysis models AI demand at 86 percent of TSMC's 2027 N3 wafer output, nearly entirely squeezing out smartphone and CPU wafers, with 2026 hyperscaler capex consensus moving materially higher and Google's roughly doubling.

Meta raised 2026 capex guidance to $125 to $145 billion, nearly double its $72.2 billion in 2025; combined 2026 capex for Meta, Microsoft, Alphabet and Amazon is estimated around $725 billion, up 77 percent; Meta signed more than 5 GW of…

IDC expects AI spending to grow 31.9 percent a year to $1.3 trillion in 2029, more than 26 percent of worldwide IT spending (IDC, Aug 2025).The chips are getting cheaper per operation.

Gartner expects more than 40 percent of agentic AI projects to be canceled by the end of 2027, naming escalating costs, unclear business value and inadequate risk controls.

MIT's NANDA group found that only about 5 percent of enterprise generative AI pilots achieve rapid revenue acceleration, based on 150 interviews, a 350-person survey and 300 public deployments; purchased tools succeeded about 67 percent of…

S&P Global surveyed more than 1,000 enterprises and found the share abandoning most of their AI initiatives before production jumped from 17 percent to 42 percent in a year, with 46 percent of proofs of concept scrapped before broad…