AMD’s take on AI: Instinct MI300X combines CPU, GPU, and 192 GB of HBM3 memory

One of the most undisputed beneficiaries of the generative AI phenomenon has been the GPU, a chip that first made its mark as a graphics accelerator for gaming. As it happens, GPUs have proven remarkably adept at facilitating and enhancing the process of training large foundation models, and executing AI inferencing workloads.

Up until now, the big winner in the generative AI GPU game has been Nvidia, thanks to a combination of strong hardware and a large installed base of CUDA software tools. However, at an event in San Francisco this week, AMD came out with new GPU and CPU hardware and important new software partnerships and updates. Together, AMD believes these developments will allow it to secure a larger share of a datacenter AI accelerator market it predicts will reach $150 billion by 2027.

AMD introduced the new Instinct MI300X chip as a dedicated generative AI accelerator. It leverages the same basic chiplet design as the previously announced Instinct MI300A (which AMD is now sampling), but the MI300X substitutes the 24 Zen 4 CPU cores in the MI300A with additional CDNA3 GPU cores and High Bandwidth Memory (HBM).

Indeed, this new chip – with a total of 153 billion transistors – boasts 192 GB of HBM and offers 5.2 TB/second of memory bandwidth. These numbers represent a 2.4x increase in memory capacity and a 1.6x enhancement in throughput compared to Nvidia’s current H100 accelerator. Although these figures may be overwhelming for most applications, large language models (LLMs) run most efficiently in memory, indicating this should translate to solid real-world performance when the chip begins sampling in the third quarter of this year.

On the software front, AMD made several significant announcements. Firstly, the company detailed the latest version of its ROCm platform for AI software development, ROCm 5. This comprises low-level libraries, compilers, development tools, and a runtime that lets AI-related workloads run natively on AMD’s Instinct line of GPU accelerators. This serves as the foundation for AI development frameworks like PyTorch, TensorFlow, and ONNX. Moreover, a new partnership with the PyTorch Foundation emerged at AMD’s event. From PyTorch 2.0 onwards, any AI models or applications developed with PyTorch will run natively on AMD Instinct accelerators that have been upgraded to support ROCm 5.4.2.