- Microsoft’s new Maia 300 chip is built for AI inference at scale
- Maia 300 ties compute, memory and networking into one AI-optimized system
- By optimizing for tokens per megawatt, Microsoft is solving for its data center power problem
Microsoft wants to run more of its own services on its in-house inferencing chips, announcing plans to scale deployments of its Maia system into multi-gigawatt territory in 2027 to handle workloads from Copilot, GitHub and Foundry. But that doesn’t necessarily mean it is looking to cut into Nvidia and AMD’s market share, as much as some would like to frame it that way.
“I don't expect Maia to slow Nvidia or AMD deployments across Azure,” Moor Insights and Strategy VP and Principal Analyst Matt Kimball wrote. “Even with massive Trainium deployments, AWS signed a $38 billion agreement with OpenAI built on Nvidia systems. AI demand is growing too fast for any single supply chain.”
So why, then, is Microsoft doing this? It all comes down to efficiency.
Microsoft announced its plans alongside the launch of its Maia 300 chip, the third generation of its custom in-house silicon. Notably, Maia boasts a software-defined dataflow architecture and AI-assisted kernel optimization to minimize data movement and improve utilization of compute, memory and networking resources. Maia 300’s 18 TBps of peak memory bandwidth is also more than double Maia 200’s 7 TBps, helping it perform better on long-context workloads and large models.
Instead of treating networking as a separate step, Microsoft has built it into the way the chip system works. That means data can move quickly between Maia chips, racks and memory, which helps large AI models run faster and more reliably at data center scale.
Futurum Research Director Brendan Burke noted that Maia 300 is built from the ground up for “long-context, reasoning-intensive, decode-heavy inference.” In addition to Maia 300’s optimizations around memory and compute, he noted Microsoft designed “a single scale-up fabric spans up to 12,288 accelerators with no separate scale-out network. Prefill and decode workers can be sized independently and KV cache moves between them on that same fabric.”
“Every one of these choices pays off in long generations at high concurrency, which is what both RL rollouts and agentic reasoning demand,” he wrote, referring to reinforcement learning.
Put it all together and Microsoft said Maia is expected to deliver 40% more tokens per megawatt than other leading GPU systems. That’s a big deal in a world where data centers are running up against major power constraints.
“New data center capacity takes years to bring online, and grid connections are scarce. Once a facility is built, its power envelope is fixed,” Kimball explained. “So, the real question is simple. How much useful AI work can you get out of every megawatt?”
Microsoft noted that it’s earlier Maia 200 system is already deployed across the U.S., Europe and Australia to serve workloads across Teams, Copilot and Cowork. The new Maia 300 chip, when in its full 96-accelerator configuration, delivers up to 7x the rack-scale token throughput that Maia 200 does, Microsoft said.
Read more about the intersection of inference, power and efficiency on Fierce Network
AI data centers could unlock more compute without more power
Chip startup Rebellions courts telcos with cheaper AI inference and an open-source pitch
SambaNova targets AI inference boom with chips built for existing data centers