- Google Cloud’s Mark Lohmeyer arguing that always-on agents will need CPUs to orchestrate access to API, apps and data and manage long-running AI workflows.
- Google Cloud is rethinking compute for agentic AI
- The rise of AI agents may reshape enterprise infrastructure, as well
After years of AI infrastructure talk revolving around GPUs, Google Cloud’s Mark Lohmeyer told Fierce CPUs are poised for something of a comeback.
The reason? Agentic AI workloads don’t behave like the chatbots that kicked off the generative AI boom. Instead of taking a prompt and returning an answer in one shot, agents are expected to spin up other agents, call APIs, access enterprise applications and data, write and run code, and preserve context across sessions that could stretch for days, weeks or even months.
“All of that requires CPUs,” Lohmeyer, who is the company’s VP and GM for AI and Computing Infrastructure, said. While inference will still lean on GPUs and TPUs, Lohmeyer said the orchestration layer around agentic workloads will drive “a ton of additional demand for CPUs.”
Lohmeyer isn’t the only one who thinks so. AMD has been banging the CPU drum for more than a year now and Red Hat also told Fierce CPUs will likely play a key role in inferencing and AI orchestration.
This shift is driving two big changes. First, the Lohmeyer said Google Cloud is designing a new class of compute services capable of handling workloads that are highly bursty, frequently idle and that need to preserve their state over long periods. These would allow customers to spin up compute when agents need it, hibernate that compute when they don’t, and restart it quickly without losing context. It’s a win-win: customers avoid paying for time when their compute is idle, while still maintaining the ability to keep long-running agentic sessions intact.
Packing more workloads onto hardware
But if you’re wondering how on earth Google Cloud – or any other provider, for that matter – can juggle many such workloads on finite compute resources, so were we. Lohmeyer said the shift toward agentic AI is also driving a rethink of how workloads are packed onto physical infrastructure (i.e. hardware). It’s all about upping utilization and maximizing resource use. Indeed, Red Hat’s Steve Watt told Fierce in May that CPU rates are notoriously low and noted the Intel and AMD-led x86 Ecosystem Advisory Group is trying to solve for this very problem.
Here’s how it could work, in theory: If an agent only needs to actively run 5% or 10% of the time, cloud providers may be able to support a high density of agentic workloads on the same physical infrastructure while keeping environments secure and separate using sandboxes.
“Basically all of those agents will run in some type of sandbox. So, it could be a VM, it could be a microVM, it could be a container that basically isolates that specific workload,” Lohmeyer explained. “We think this actually becomes really, really important for the industry because as the demand continues to rise, you want to make the most effective use of those underlying compute and memory and other resources. That ability to consolidate, let's say 10 to one or higher we think is going to be really important to meet the scale that our customers expect.”
Though Lohmeyer didn’t mention the Advisory Group’s work on CPU utilization, he noted during the conversation that Google Cloud has deep partnerships with both AMD and Intel to run x86-based instances, alongside Google’s own Arm-based Axion CPU instances.
Not quite a new chip
Lohmeyer isn’t ready to say agentic AI demands an entirely new class of CPU. However, he did note the workload is different enough that cloud providers will need to rethink how CPUs, storage, networking and orchestration software are designed together.
He described co-design as one of the more important principles for the agentic era — not just tuning the CPU itself, but optimizing the hardware and software stack around workloads that can hibernate for long stretches, preserve state outside active memory and then restart en masse when demand returns.
“How it flows down into specific chip architecture is probably a little bit early for that, but I think this idea of codesign is going to be important just like it has been for GPUs and TPUs and the software stuck on top of them,” he said. “The same thing I think will be true here.”
The implications for enterprises
As enterprises hustle to get compute ready for the agentic future, Lohmeyer said they need to be thinking about more than just what it takes to run the agents. They need to make sure their other infrastructure – APIs, data sources, interconnectivity – is up to snuff.
“Yes, you need the optimal infrastructure for the agent itself. But then that has repercussions for the rest of the applications and data and the infrastructure that powers those in the enterprise, because it's going to be accessing it with different types of patterns than we've seen in the past,” Lohmeyer said.
That, Lohmeyer concluded, will force enterprises to rethink not only how they scale those back-end platforms, but where they locate data and applications so they sit closer to where the agents are running.
Read more about CPUs and AI on Fierce Network: