"Father of HBM": Future AI Competition Hinges on Memory, 3D Architecture Decided by Cooling and Power Supply, China and U.S. Will Catch Up

Kim Jung-ho, a professor at the Korea Advanced Institute of Science and Technology (KAIST) hailed by Korean media as the "Father of HBM," passionately stated that the essence of AI is memory.

In an exclusive interview with The Dong-A Ilbo, he presented this disruptive view, pointing out that the industry has long overemphasized model algorithms and GPU computing power. However, when large models run in practice, every stage—from attention mechanisms and KV caching to inference generation—relies heavily on memory system support.

Kim explained that HBM (High Bandwidth Memory) dramatically expands the number of data transmission "lanes" by vertically stacking multiple layers of DRAM. He vividly compared it: "Traditional memory is like an 8-lane highway, while HBM is like 1024 or even 2048 lanes. In the future, it could evolve to a million lanes."

Yet, increased bandwidth alone does not eliminate bottlenecks. During inference, large models frequently read and write data from HBM or future HBF (High Bandwidth Flash), causing GPUs to spend most of their time idle, waiting for data to arrive.

Kim estimates that even with a million GPUs deployed, only about 10% of their time is actually used for computation. Even with algorithmic optimization, this figure is unlikely to exceed 30%.

To address the energy consumption and latency caused by data movement, Kim predicts that future AI computing will shift toward a "memory-centric" architecture.

He likens the ideal 3D structure to a 100-story building: "Place the GPU on the first floor—data just takes an elevator down for computation, and everything is handled within the building, eliminating long-distance travel." Starting with HBM4, some GPU functions will be integrated directly into the memory structure.

Kim further outlines a three-tier heterogeneous memory system: ultra-fast HBM (stacked DRAM) acts as a department store for hot data, high-capacity HBF (stacked NAND) functions like apartment complexes for cold data, and his proposed HBS (High Bandwidth SRAM) serves as an ultra-low-latency cache. GPUs and CPUs will be placed on the top layer, focusing solely on heat dissipation and task scheduling.

Kim specifically notes that NVIDIA CEO Jensen Huang's frequent visits to Korea stem from the fact that GPU microarchitecture performance gains are nearing stagnation, and their high heat output makes them difficult to stack at scale like memory chips.

He believes that in the future, the power delivery and cooling capabilities of 3D AI computers will become the core competitive advantage, determining corporate survival. This suggests that companies like Samsung and SK Hynix, with deep expertise in both DRAM and NAND technologies, could secure a central role in the next-generation architecture.

However, Kim warns this is not simply a shift from the "GPU era" to a "memory era," but rather a system-level race involving GPUs, HBM, advanced packaging, interconnect networks, and thermal management.

Facing growing pressure from Chinese firms aggressively developing their own GPUs/NPUs and advancing domestic memory technologies, Kim candidly told the Korean industry: "This is a long-term survival race—we must strive to survive until the end. "