CPUs are the agent bottleneck

For three years the unit of AI capacity has been the GPU. Budgets, power contracts and data-centre designs were drawn around accelerators, with a handful of host CPUs to feed them. That made sense when the workload was training, or a chatbot answering one prompt at a time. It makes less sense for agents.

An agent alternates between thinking and doing. The thinking is a model call. The doing is everything else: running a search, executing generated code in a sandbox, driving a browser, parsing a PDF, calling an API, writing files, deciding what to try next. Almost all of that is general-purpose CPU work, and in many agent workloads it is where the time goes.

The measurements

The clearest evidence comes from a profiling study first posted to arXiv in November 2025. Its authors ran five representative agent workloads, including retrieval-augmented generation, a chemistry agent built on tool use and a software-engineering agent, and broke down where latency and energy were spent. They found that CPU-side tool processing accounted for up to 90.6% of total latency, and that CPU dynamic energy reached up to 44% of the total at large batch sizes. They also found that throughput could be limited by either side: by CPU effects such as coherence, synchronisation and oversubscribed cores, or by GPU memory capacity and bandwidth.

The point is not that GPUs stop mattering. It is that, for an agent, a fast model attached to a slow or overcrowded host spends much of its time waiting.

What the chip makers say

Vendors with every reason to sell accelerators are now saying the same thing.

Intel's chief financial officer, David Zinsner, told analysts on the company's first-quarter 2026 call that training systems generally run at eight GPUs to one CPU, that inference is "probably getting into the 3 or 4 to 1 kind of level", and that agentic and multi-agent work could potentially flip the ratio the other way. Intel sells CPUs, so the claim should be read as a vendor's view, but it is a specific one.

NVIDIA, whose business is the accelerator, introduced Vera in May 2026 as the CPU for agents. Its announcement names agent tool use and sandbox execution as CPU work on the critical path of an AI data centre, and argues that the measure is shifting from cores per dollar towards tokens per dollar, which rewards CPUs that finish agent and orchestration work faster. Its performance figures are its own and not yet independently verified; the direction of the argument is what matters here.

From Taiwan, the market research firm TrendForce puts today's AI data centres at roughly one CPU to every four to eight GPUs, and expects agent deployments to move that towards one to one or one to two. It cites an estimate from Arm, the UK chip designer, that CPU cores needed per gigawatt of AI capacity could rise about fourfold, from roughly 30 million to 120 million. TrendForce also reports that Intel and AMD raised prices on selected CPU lines towards the end of the first quarter of 2026, as server CPU supply tightened.

These are forecasts and vendor statements, not settled measurements, and they disagree on the exact ratio. They agree on the direction.

Why agents behave this way

Three properties of agent workloads push demand onto CPUs.

Agents run code. Every tool call that executes generated code needs an isolated environment, and isolation has a cost in cores and memory per session. AWS, for example, runs each AgentCore session in a dedicated microVM with its own CPU, memory and filesystem. Multiply that by every concurrent agent.

Agents hold state. A sandbox that stays alive while an agent waits for a person, a page to load or a long build keeps its cores and memory reserved, whether or not the model is busy. Sandbox services expose this directly in their limits; Modal's sandboxes have a default lifetime of five minutes and a maximum of 24 hours, after which state has to be snapshotted and restored.

Agents fan out. One request can trigger many tool calls, browser sessions or sub-agents. Orchestration, scheduling and data movement between them all run on the host. The arXiv study's finding that oversubscribed cores and synchronisation can cap throughput is a description of exactly this.

What this means for CTOs and researchers

Size for sessions, not tokens. A capacity plan built on tokens per second per GPU will under-provision the hosts. The better unit is concurrent agent sessions: cores, memory and sandbox overhead per session, multiplied by how long sessions stay open, with model capacity sized alongside.

Expect the ratio to move, and measure your own. Published ratios range from one CPU per eight GPUs to parity and beyond. Your own tool mix decides where you land. Profile a representative agent end to end and record how much wall-clock time is spent outside the model before buying hardware.

Treat CPU supply as a risk. If agent demand is tightening server CPU supply, as TrendForce reports, lead times and prices for host capacity will matter as much as accelerator allocations.

Keep the GPU busy by keeping it fed. The arXiv authors show scheduling changes that overlap CPU and GPU work can reduce latency substantially. Batching, admission and scheduling on the host are worth engineering effort, because idle accelerators waiting on tools are the most expensive idle time in the building.

For researchers, the open questions are practical: benchmarks that report end-to-end agent latency rather than tokens per second, sandbox isolation that costs fewer cores per session, and schedulers that treat the CPU and the accelerator as one system.

Where Praecise fits

Praecise Cloud runs agents, MCP servers and backends on hardware Praecise owns, built from source, placed behind the edge and supervised. Praecise Metal operates those machines to measured targets for placement, power and thermal control, and offers GPU capacity to infrastructure owners who need it. Both run on Praecise Furcate and the open-source Furcate operating system for machines, edge cells and data centres. Praecise Engine, open source under Apache-2.0, sits between an application and an inference backend and makes token generation faster through speculation policy, request shaping and admission.