The recent HPC + AI on Wall Street show presented a panel that brought together key vendors to discuss the current GPU Squeeze (shortage).   The “Squeeze” is due to the rapid and continuing rise of Generative AI/LLMs and has now pushed the demand for GPUs to unprecedented levels and created a “squeeze” on GPU availability.

A full 40-minute panel video is available for viewing. (In addition to all the other talks and panels of the event.) The panel was designed to provide views from several vantage points, including hardware, cloud, and on-prem perspectives. The was moderated by myself and consisted of the following members:

• Wyatt Gorman – Solutions Manager, HPC and AI Infrastructure, Google Cloud
• Prabhu Ramamoorthy – Global Partner Success Manager, Nvidia
• Thomas Jorgensen – Senior Director, Technology Enablement, Supermicro
• Kiran Agrahara – Cloud Solutions Architect, Intel Corporation

The panel covered four topics: HPC and GenAI, The GPU Shortage, Alternatives, and HPC/GenAI hardware convergence. A synopsis of the panel’s discussion is provided for each question. The full discussion is available in the video.

Question 1: HPC and GenAI
Some users can find the line between traditional HPC and GenAI a little blurry. The need for fast GPUs indicates both types of applications are doing lots of number crunching, but is a traditional HPC Monte Carlo risk analysis similar to a GenAI trained to do risk analysis? In other words, how do you see GenAI and HPC? Are they in the same broad category or represent two distinct markets? Do they complement each other?

NVidia’s Prabhu Ramamoorthy mentioned right away that he sees this convergence happening now, and they see clients doing a mix of both HPC plus AI solutions as they are working toward their end use cases.

Google Cloud’s Wyatt Gorman replied that he sees people bringing machine learning techniques into HPC processes at different levels. So consider it a subset, a domain within HPC. He suggests it might spin out and become more siloed like Big Data may have out of HPC. But for now, he thinks it’s converging.

Question 2: The Shortage
Based on the news we have reported, I assume there are some challenges for HPC users accessing GPUs (purchased or in the cloud). I want to check my assumption and ask from your company’s perspective how big an issue is the “GPU Squeeze.”

Thomas Jorgensen of Supermicro shared an interesting data point. “I can tell you that we have tens of thousands of systems on backorder because of the lack of H100 GPUs. But, I will also say that there are a number of alternatives on the market, and we have a group of newer CPUs, for instance, for HPC workloads, which traditionally has been serviced by GPUs.”

Thomas mentioned the new Intel Max Xeon CPUs with 64 GB of embedded HBM2 memory, greatly enhancing some of the HPC workloads. He suggests that on-prem HPC is one of the ways to alleviate the situation if you cannot get Nvidia GPUs. As a hardware-neutral vendor, he also mentioned other GPUs from both Intel and AMD are easily available.

Thomas also mentions an important point, “A lot of people are saying we are dependent on CUDA, but if you have to wait a year for NVidia GPUS, you can convert a lot of code in that period. So there is absolutely life in some of the other GPUs available on the market, and some of them can be hacked with very little wait time.”

He concluded by also mentioning the Nvidia L40 GPUs, which he sees as a great replacement right now or at least something you can purchase. In addition, Thomas mentioned, “So, I will say for our on-prem customers, there’s a lot of alternatives, and customers are exploring that right now. And to great success, I would add.”

Kiran Agrahara from Intel added, “It depends on the use case. When you look at the shortages, it depends on what use case you’re looking into.” He continued, “There are alternatives available, and most of the learning and inferencing can be done with Xeon Gen 4 processors today.” He mentioned Intel has benchmarks available for different use cases. He also pointed out that everybody thinks an NVidia GPU is needed for every use case.

Kiran also pointed out that Intel has Gaudi 2, which is more of an inferencing processor than a learning processor and is available on-prem and in the cloud. Continuing, he mentioned Datacenter GPU Max Series that could be a substitute or an alternative to A100s. He concluded by calling attention to Intel’s OpenVINO (Open Visual Inference and Neural Network Optimization) — a software layer on the top of all the Intel HPC and AI products.

Prabhu Ramamoorthy pointed out that people think that this GPU shortage happened recently, but we have seen it for several years. For instance, the GPUs ran in the cloud three years ago doing NLP (Natural Language Processing). A lot of hedge funds were running all these workloads. Generative AI is like the new, beautiful, sexier term for NLP, and now there is much more demand for these GPUs. We want customers to choose the right use case –as pointed out by Thomas. He also reminded everyone that NVidia employs more software engineers than hardware engineers. As a result of these efforts, he mentioned Nvidia software like TensorRT-LLM that optimizes applications and ensures that GPUs are utilized more efficiently.

Wyatt Gorman concluded that he thinks it’s important to plan your GPU needs for the long term. Now, look at alternatives and not get too far ahead of your applications. Consider what possibilities are out there now, and there are ways to optimize GPU use, and that are good alternatives. are good alternatives.

Question 3: GPU Alternatives
Reports of the shortage may last another 18 months. Are there alternative ways to suggest HPC users run applications (cloud, alternative hardware, software optimizations)? For instance, NAMD, an HPC molecular dynamics code, will run using CPUs and MPI and/or with GPUs. Will users revert to MPI?

Kiran Agrahara recalled an older example where users needed 20 GPUs, and for the same work to be done, they needed about 180 CPUs. But he suggested looking at what’s changed in the last three years. He continues, “The CPUs have come a long way. Back when we did not have hardware accelerators. If you look at the next-generation Xeon processors, we have an advanced matrix architecture called RMX. So today, the CPUs have come a long way. You’re looking at 128 cores, and then you have RMX architecture, which I can say is like AVX512 on steroids. So the answer is yes, you can accomplish this with CPUs.”

Wyatt Gorman added that it is really important to make people aware of smaller GPUs. Everyone is thinking about A100s and H100s right now. The slower Nvidia T4, L4, and L40 GPUs have the capacity and are available on Google Cloud because, right now, they aren’t in such high demand. If you do some tuning and optimization for these GPUs, you can see the acceleration of your code.

Question 4: HPC and Data Center Convergence
Typically HPC is in its own kind of silo, and then you have rapidly growing GenAI with similar hardware in another developing silo. Do you see that this may benefit HPC when people start to recognize we could bring both of these resources together? What do you think about the convergence of HPC applications running in the data center versus in the R&D lab with separate hardware?

Wyatt Gorman shared that Google sees more AI capability in HPC RFPs. He thinks as these problem types, as HPC and AI converge, we see AI techniques being brought into traditional HPC problem-solving applications, and we’ll see more and more of that and more and more resources being necessary. As I mentioned, Google TPUs are now supported in Slurm, and you don’t necessarily need Kubernetes to run HPC now or in the future. And you know, it won’t be necessary even when it’s an option. 

Thomas Jorgensen of Supermicro mentioned that the Intel fifth-generation roadmap showed up to 288 cores on some future CPUs. There are absolutely things also happening on the CPU platform that will result in more performance than a traditional CPU platform. Another point is the Max GPU that Intel has created is an HPC plus AI GPU. The first test we’ve done at Supermicro shows real performance for HPC with that GPU.

From a hardware standpoint, Supermicro tries to be Switzerland. We work with Nvidia, AMD, and Intel.

Thomas continued, “Like Intel AMD platforms, we have very high core count Epyc processors and a science experiment with Nvidia. In this test, we can place ten GPUs in a single system. The HPC benchmarks, and the performance we are getting out of one system on HPC is astounding. But it’s a way forward, at least for, I mean, it doesn’t help with the squeeze, right, because using H100, but there’s some performance there that shows real promise for HPC workloads. So I’m very hopeful for the hardware and some optimizations we’re seeing that can wring more from existing hardware.”

As the moderator, I should note that after the panel concluded, the panelists continued this discussion in the speaker’s green room for about 20 minutes. The market is not done with this topic.

These above responses are a summary of the full GPU Squeeze panel. There is a lot more insightful discussion to be found on all the videos.