Nvidia Introduces New AI Chip, the HGX H200, with Improved Memory Capacity and Bandwidth

Nvidia is preparing to launch a cutting-edge AI chip, the HGX H200, designed to enhance the performance of demanding generative AI tasks. This new GPU boasts 1.4 times more memory bandwidth and 1.8 times more memory capacity than its predecessor, the H100, which has been in high demand. There is some uncertainty surrounding the availability of the new chips, as there may be supply constraints. Nvidia intends to work with global system manufacturers and cloud service providers to ensure the H200 is accessible. The initial shipment of H200 chips is scheduled for the second quarter of 2024.

Enhanced Memory Performance

The H200 chip shares many similarities with its predecessor, the H100, except for its memory capabilities. The H200 features a new, faster memory spec called HBM3e, which boosts its memory bandwidth to 4.8 terabytes per second, up from 3.35 terabytes per second on the H100. Additionally, the total memory capacity of the H200 is increased to 141GB, compared to the 80GB of the H100. These memory enhancements contribute to accelerated performance for computationally demanding tasks, including generative AI models and high-performance computing applications.

Compatibility and Availability

The H200 is designed to be compatible with systems that already support H100s, meaning cloud providers won’t need to make any changes when integrating the new chips. Leading cloud service providers such as Amazon, Google, Microsoft, and Oracle are expected to be among the first to offer the H200 GPUs next year. However, Nvidia has not disclosed the pricing details for the new chips, which are anticipated to be expensive. The prior-generation H100s are estimated to sell for anywhere between $25,000 to $40,000 each, with multiple chips required for high-level operations. Further information on pricing and availability is yet to be revealed by Nvidia.

Increasing Demand for AI Chips

Nvidia’s announcement comes at a time when AI companies are desperately seeking H100 chips. Nvidia’s chips are highly regarded as the best option for efficiently processing the vast amounts of data needed for training and operating generative image tools and large language models. The scarcity of H100 chips has led companies to use them as collateral for loans. Startups have even resorted to collaborative efforts to gain access to these chips. While Nvidia plans to triple the production of H100 chips in 2024, the demand for generative AI remains high, potentially driving even greater demand for the new H200 chip.