- Agentic AI is driving CPU demand, but SambaNova noted they're doing the same work as before – just more of it
- Nvidia argued agentic AI benefits from faster CPU cores and higher instructions per clock to accelerate the loop between CPUs and GPUs
- Google Cloud emphasized hardware-software co-design, saying it is too early to determine the implications for specific CPU architectures
CPUs are having a bit of a comeback moment. Executives from AMD, Google Cloud and Vultr have told Fierce that agentic AI is driving demand for CPUs to orchestrate all the actions agents take. But a point of tension has bubbled to the surface: does agentic AI simply require more CPUs or specialized ones?
The answer, of course, depends on who you ask.
Anton McGonnell is VP of Product and Core Tech at chip company SambaNova, which has made a name for itself with its inference-focused reconfigurable dataflow units (RDUs). The company has close ties with Intel, having developed a reference architecture that incorporates SambaNova’s RDUs with Intel Xeon 6 CPUs.
McGonnell told Fierce that “the profile of what an agentic CPU needs relative to CPUs that came before it aren’t necessarily very different.”
While GPUs and RDUs hand the actual read and write cycles for inference (aka, pre-fill and decode), CPUs essentially act as the orchestrators of the agentic world, handing all the tool calls that agents need to make to get their jobs done. Think querying databases, compiling and executing code, processing images, etc. So, they are essentially doing the same job they used to do for humans but now for agents.
“It’s not new work. It’s the same work computers already do. It’s just a lot more of it. That’s the key,” McGonnell said. “That’s the reason CPUs have made such a big resurgence.”
McGonnell added there’s not a huge need for a CPU redesign because “the offload between the inference chip at the prefill or decode stage to that tool call that’s running on the CPU is not super latency sensitive.”
Nvidia’s case for faster CPU cores
But Nvidia Cloud and AI Product Marketing Leader Hannah Coutand made a very different argument. Nvidia, of course, has been expanding its presence in the CPU market with its Grace and Vera chips.
According to Coutand, Grace was an “off-the-shelf ARM design.” Vera, though, was intentionally designed to be the best host-node for Nvidia’s GPUs as well as a standalone CPU. The company, she said, focused on both high per core performance and efficient throughput scaling with Vera.
Why? Because Coutand explained that speeding the information loop between CPUs and GPUs in the AI era matters.
“Our goal is to make that loop as fast as possible because when you’re agent is working on the CPU, your GPU is holding that context, holding that KV Cache in there. So you want to be able to return as quickly as possible,” she said. “The importance of the CPU in that loop is only going to get more important.”
Coutand said the answer to better performance used to be adding more compute cores. But adding more cores doesn’t necessarily solve for speed at the individual core level. For that, you need to push for higher instructions per clock.
“CPU design is really all about trade-offs. So back in the hyperscale growth period, they were trying to sell virtual CPUs and in order to sell virtual CPUs they were like ‘well, just give me a bunch of cores. I don’t care how fast those cores are, I just want a lot of cores,’” Coutand said. “This agentic AI movement has actually really flipped the whole dynamic on its head,” prompting companies to seek not just more cores but faster ones.
Google Cloud focuses on co-design
In a recent conversation with Google Cloud’s VP and GM of AI and Compute Infrastructure Mark Lohmeyer, Fierce asked whether the company sees a need to design a new kind of CPU specifically for agentic workloads. Lohmeyer said this question is one the industry is actively exploring.
“Agentic workloads are relatively new on the scene, so we’re working through what’s the best way to support that,” he said. “From Google’s perspective, I would say one thing we think is really important is the idea of co-design, where effectively you’re codesigning across the software and the hardware.”
This might allow, for instance, agents to rapidly hibernate and store their state in a disk as opposed to in memory on the CPU. “Those are the types of things that we’re looking at very actively. How it flows down into specific chip architecture, it’s probably a little bit early for that,” he concluded.
Read more about the AI-driven CPU comeback on Fierce Network:
Red Hat exec: Sovereign AI could drive CPU demand for inference