- Fierce talked to Nvidia’s SVP of Networking Gilad Shainer, who explained the genesis of the company’s networking stack
- He said Nvidia’s networking is based on technology from Mellanox, a company Nvidia bought in 2020
- Shainer said there are actually five different kinds of networks necessary for an AI factory
OCP GLOBAL SUMMIT, SAN JOSE, CALIFORNIA — Nvidia is famous, obviously, for its GPUs, which have become the computing backbone for AI. But Nvidia’s executives also had foresight several years ago — prior to ChatGPT — that networking was going to be an important factor to connect all the GPUs together.
At this week’s OCP conference, Fierce Network caught up with Nvidia’s SVP of Networking Gilad Shainer, who explained the genesis of the company’s networking stack.
Shainer previously worked for a company called Mellanox, whose InfiniBand technology provided high-speed networking to connect compute and storage systems, especially in high-performance computing data centers. Nvidia acquired Mellanox in 2020 and has since made InfiniBand a major part of its AI data-center networking portfolio.
Of Nvidia’s foresight, Shainer said, “They understood that it's not enough to be a GPU company. It's a computer company, and that means that you need to build a full data center.”
ChatGPT rolled out commercially in November 2022, but OpenAI had been working for years before that, and it needed GPUs for all its initial training.
When Nvidia announced in 2019 it was buying Mellanox for $6.9 billion, Nvidia CEO Jensen Huang said, “The emergence of AI and data science, as well as billions of simultaneous computer users, is fueling skyrocketing demand on the world’s data centers. Addressing this demand will require holistic architectures that connect vast numbers of fast computing nodes.”
Shainer said AI workloads require multiple GPUs, and these GPUs need to work essentially as a single unit because everything needs to be synchronized.
Of AI workloads, he said, “If I have a big problem, I need to split it into small pieces. I will calculate the first step. And then I need to aggregate the result. And now that's going to become my first step again. Now, let's assume that I don't have a great network. Each GPU will get the data in a different time. I cannot go to the next phase before all of them finish their first step, and I combine the results.”
So, it’s critical that the GPUs work in synchronization to not waste precious computing time. “That means you need to have a special network,” said Shainer. “You cannot have fluctuations in time."
Vera Rubin
At OCP Nvidia was touting its Vera Rubin stack. Vera is Nvidia’s CPU architecture, and Rubin is its GPU architecture. The Vera Rubin NVL72 is a combined rack-scale system, containing 72 Rubin GPUs and 36 Vera CPUs. The GPUs are connected with a network called NVLink, and NVLink is considered a scale-up network (server-to-server).
Nvidia then connects a bunch of Vera Rubin racks together because to run training models requires “hundreds of thousands of GPUs, not just 72,” said Shainer. This is considered scale-out infrastructure (rack-to-rack).
Other networks for AI factories
In addition, there are other networking needs in AI data centers. Shainer said there are actually five networks for AI factories. In addition to the scale-up and scale-out networks, there is often the need to connect different data centers together because there might not be enough power available in just one location. “So this is where I need a network that will be able to connect long distances, but make those two data centers behave like one,” said Shainer. “So that's what we call scale-across.”
Then there’s a separate network for storage, which is necessary for inference. And Shainer called this a context-scale network for inferencing.
“And you also need to have a scale-in infrastructure that enables secure access of users to the AI factory and to enable security across all agents running within the factory,” he said.
Whew! That’s a lot of networks for an AI factory.
Fierce asked about the physical material for all these networks, and Shainer said, “Copper is great for short distances. Using copper to connect networking devices is the best thing ever because it's very cost-effective. It's very reliable, and it consumes almost no power.”
For that reason copper is used for short-distance connectivity within racks and connecting racks. But to connect anything further apart, then AI data centers use fiber optic connectivity.
Read more about Nvidia's networking moves on Fierce Network:
Cisco and Nvidia develop new pluggable optic module for AI data centers