AI’s next big bottleneck may not be compute — it may be memory. As enterprises move from training massive models to running inference at scale, the infrastructure challenge is shifting from simply getting enough GPUs to keeping those GPUs fed with data fast enough to make the investment pay off.
In this episode of The Five Nine, we dig into the memory squeeze with Nvidia’s Jason Hardy and WEKA’s Val Bercovivi, looking at why AI workloads are becoming memory-bound, how agents and KV cache are changing infrastructure economics, and why a GPU cluster optimized for training may not be the right answer for inference.
From high-bandwidth memory and storage tiering to GPU Direct Storage and cache-read token pricing, the conversation unpacks what has to change as AI moves into its next phase. The big question: can companies make inference profitable without overbuying the world’s most expensive compute?
Catch the video at top, listen to the audio edition and read our transcript below, or watch this and future episodes on YouTube.
This podcast is written and hosted by Diana Goovaerts. It is edited by Diana Goovaerts and Matt Rickman. Liz Coyne is our executive producer. Special thanks to guests Jason Hardy and Val Bercovici.
To learn more about the topics in this episode, check out:
Diana Goovaerts: AI's next big bottleneck may not be compute. As models get bigger and agents multiply, the pressure is shifting to memory, and that's forcing a rethink of inference infrastructure, storage, GPU utilization and AI economics.
Val Bercovici: When you take a look at the combination, the popular combination of big new models with popular coding agents, research agents, security agents we're just simply out of memory.
Anytime you want to do something very valuable in AI, you become memory bound now before GPU bound.
Diana Goovaerts: I'm Diana Goovaerts, and this is The Five Nine.
For the past couple of years, the AI infrastructure story has been dominated by compute. Who has the chips? Who can get more of them? Who can build data centers fast enough to power them? But as AI shifts from training to inference and from single prompts to armies of agents, the bottleneck is moving.
Today, we're talking about memory, what it is, why AI is consuming so much of it, and why the industry's obsession with GPUs may be missing the next major constraint.
Jason Hardy: Data has become the bottleneck. It's delivering data to the GPUs to be able to continue to create those tokens and manufacture that intelligence out of it.
Diana Goovaerts: Before we get to the squeeze, it helps to define what we mean by memory because it's not just one thing. There's high bandwidth memory attached to GPUs. There's system memory. There's NVMe flash. There are storage systems. And all of those tiers matter because AI needs to constantly pull data in, process it and create new data on the way out
Jason Hardy: So memory's – it's in everything, and there's obviously various types of memory. So we have memory in HBM, a part of the GPU. That's considered memory. We have storage devices like NVMe, flash storage, SSDs. That is a type of memory. System memory, which is like the RAM or the memory in your laptop or your desktop or in the GPU servers. That's all a different type of memory.
And there's different pieces of memory, different memory technologies that do different things and support different types of workloads.
Diana Goovaerts: It's worth zeroing in on that point. AI isn't just consuming existing data, it's generating more of it. Agents in particular can create long chains of prompts, responses, intermediate work and cached context. Each step adds pressure to the memory hierarchy.
Jason Hardy: We are consuming a tremendous amount of information and generating a lot more information from it.
So yes, there is a output that comes from working with AI that does consume data. KV cache is a topic obviously, too, as we're inferring and interfacing with the LLMs. It creates data around it as we're working with the prompts and the different agents.
Diana Goovaerts: And that's where memory matters. The big shift here is inference. Training is still enormously expensive, but it is not the same workload as serving AI answers to users in real time. Val put it bluntly: a cluster that works well for training may not be optimized for inference at all.
Val Bercovici: Just because you buy a GPU cluster from NVIDIA that's fantastic for training, and the market has proven that over the years, does not mean the same cluster or the same network and storage and memory configuration is optimal for inference. In fact, it's almost always not.
Diana Goovaerts: And why is that? Because inference has phases. There's a compute-heavy prefill phase where the model takes in the prompt and builds up the context. Then comes decode, where the model generates the output tokens. And that second phase is, in Val's words, where the memory wall shows up.
Val Bercovici: Inference is very, very memory bound. Inference actually has, let's say for simplicity, two phases.
Once you finish that first phase, the pre-fill phase, which burns a lot of energy, consumes a lot of the compute side, the tensor cores of GPUs, for example, then you're not done. Then you just begin the work of actually seeing your output tokens, and that phase is called decode. And you're basically transferring memory back and forth at that phase. You're doing very, very little processing.
That's where we're seeing the memory bottleneck, the memory wall, because we do all this work and we don't see any output yet until we go through this phase. The faster you can transfer that KV cache and move that memory fundamentally at higher bandwidth, the faster the reasoning tokens happen before output, and of course, the faster the output tokens happen.
And this is compounded for agents because it's not one session, if you will, it's multiple sessions, hundreds of subtasks.
Diana Goovaerts: The memory squeeze is not just a technical issue either. It's an economic issue. GPUs are among the most expensive pieces of infrastructure companies can buy, and if those GPUs are waiting around because data can't get to them fast enough, that capital is not working as hard as it should.
Jason Hardy: We want this investment, these GPUs, to operate as efficiently, as effectively as possible. If I bought this thing and I want it to continually manufacture intelligence, I need to be able to keep it busy.
So I need to have systems that can continue to feed it as those requests come through so that we're not idle or we're not stalling while we're waiting for information.
Val Bercovici: So you can basically over-provision GPUs to solve this problem because you're sizing for memory and leaving a lot of that processing idle, especially in that second decode phase. Or you can try and run a profitable, you know, positive gross margin business and not over-provision this extremely expensive, hard-to-find infrastructure, and that's where you need to be very, very efficient with your memory and have proper memory tiering implemented.
Inference is how you recoup that investment, how you make money back.
Diana Goovaerts: That is at the heart of the spending question. If companies keep solving a memory problem by buying more GPUs, they may end up over-provisioning compute just to get more memory attached to it. That can work, but it's really an expensive way to solve the problem.
So what are the fixes? Well, one answer is memory tiering, which means using different layers of memory and storage more intelligently instead of assuming that everything has to sit in the most expensive memory that is attached to the GPU.
Val Bercovici: You need different networking setups for those GPUs, a lot more memory for those GPUs, and the topic that's very timely right now is how you can offload some of that memory to other memory tiers, because the most scarce, expensive memory, we joke 'the most expensive real estate in the world,' is really those memory chips on the GPU package itself.
Once you're out of that very, very quickly due to big models and agents, you have to tier to another form of memory, the more common CPU dynamic RAM, DRAM, as opposed to the high bandwidth memory on the GPU itself. That is actually a very sophisticated set of leading-edge software that is constantly updated to keep up with the changes in the models themselves.
And a very trendy thing right now is when you're also out of that memory tier to go ahead and tier waterfall, if you will, to that next tier of memory, which is a form of high-speed storage. Regular storage, regular data center storage does not apply here. It has to be extremely high-performance storage.
Diana Goovaerts: Another is finding faster paths between storage and GPU memory, so data doesn't have to bounce through a CPU and system memory first.
That's what NVIDIA has been working on with cuFILE.
Jason Hardy: We call it GPU direct storage. cuFile is the library that we use for POSIX file systems, if we can nerd out a little bit here.
But the fact that we can have a GPU request data from a system and have it route directly to GPU memory, skipping that bounce buffer in the middle and having to hop through CPU and system memory and going straight to – so instead of a multi-hop, it's like we can go straight to it – that makes a huge difference because now you don't have to design in all these system memory requirements and all these like, wait a minute, I've got to stage memory in, and things like that.
Diana Goovaerts: The key point is that regular storage is not enough. The storage tier has to be fast enough and connected in the right way to act more like an expansion of memory than a traditional data repository. And agents make this whole problem bigger because they are not just answering one question and then stopping.
They can run multiple tasks over long time horizons and with many rounds of reasoning and context in between, and that increases the amount of memory needed to keep the work moving.
Val Bercovici: We have this concept of long horizon agents, which means instead of one request and response, it's thousands of requests and responses.
Instead of happening instantly or over the span of a minute, it's over hours and days and weeks now.
Diana Goovaerts: And that helps explain why the memory debate is becoming inseparable from the tokenomics debate. If an agent runs longer, does more work, and relies more heavily on cached context, then the cost of reading and moving and reusing the cache becomes central to whether inference is profitable.
Val Bercovici: Inference costs are wonderfully transparent. There are prices for input tokens. There's usually a dollar or cents price per million input tokens. Transparent similar pricing, which are usually higher for output tokens, and the most important one is price per cache read tokens.
80, 90% of all agent tokens today are those cache read tokens. It's a very competitive playing field in terms of who can offer the best pricing, and the sharp eye focuses on that cache read pricing because the regular input, the regular output pricing for an agent almost becomes a rounding error. That's the competitive front right now, and that pricing is almost a direct function of this thing called the KV cache hit rate. How often are you actually hitting that KV cache you spent a lot of money to create for every prompt session?
Diana Goovaerts: If there is one takeaway from these conversations, it is this: AI infrastructure is entering a new phase. The first phase was about getting enough compute, and the next phase is about keeping that compute fed. That means memory, storage, networking and software all have to evolve together, especially as inference and agents become the dominant workload.
That is all the time we have for today. Thank you so much for listening to The Five Nine, and make sure you like and subscribe wherever you listen or watch. Stay tuned to fierce-network.com for news in between episodes, and otherwise, we will catch you next time.