For two years, every conversation about AI capacity came back to one word: GPUs. The scarce thing, the expensive thing, the thing enterprises queued months to secure. That story is now out of date. SK Hynix, the company that supplies the high-bandwidth memory bonded to Nvidia's chips, just reported an operating margin of 72%, higher than Nvidia's own 65%, and its shares have climbed roughly 770% in a year. The binding constraint on AI has quietly moved one layer down the stack, and most enterprises are still budgeting as though it sits where it did in 2024.

The evidence for that shift is concrete. Memory now accounts for roughly 30% of the bill of materials for an AI server, up from a small fraction a few years ago, and on the highest-end configurations the share is higher still. When DRAM contract prices jump 80% to 95% in a single quarter, as they did at the start of 2026, that increase flows straight into the cost of every server built. The effect has reached the top line of the whole industry: rising memory costs were a specific, named reason the 2026 forecast for global data-center capital spending was revised upward past a trillion dollars. And there is no relief valve, because HBM output is already committed through 2026. The cost base has moved, and the supply that would relieve it is spoken for.

What Kind of Memory Is Actually in Short Supply

"Memory" is not one component. A modern AI system relies on several distinct types, each doing a different job, and only some of them are in crisis. Knowing which, and why, is most of the story.

SRAM (static random-access memory) is the small, ultra-fast memory that feeds the compute cores directly. It holds the specific data the chip is calculating with at that moment, the slices of weights and intermediate results moving through the arithmetic units, staged right at the edge of the processing element. It is the fastest tier but also the most expensive and least dense, so there is only ever enough of it to stage the immediate working set, never enough to hold a whole model.

DRAM (dynamic random-access memory) is the high-capacity working memory that holds the model’s parameters and its live computational state. It gives up some of SRAM's speed in exchange for far greater density at lower cost, which is what lets it keep billions of parameters on hand at once.

NAND flash is the storage tier. It keeps data permanently, even with the power off, at a fraction of DRAM's cost, but it is on the order of a thousand times slower to read than working memory, far too slow to feed the chip during computation. Its contribution is persistence: it acts as the warehouse that holds the model files and their billions of parameters at rest, the training datasets, the checkpoints that let a training run resume after a failure, and the vector databases that retrieval-augmented systems search before a model answers.

The layer of memory that has recently been the chokepoint in the AI system architecture is a type of DRAM called HBM (high-bandwidth memory), a version built to move data into the chip fast enough to keep the compute cores fed. HBM is where a running model actually lives during inference: the model's parameters and its context cache both sit here, and its job is to stream them to the processor at enormous speed. For large-model accelerators, no other form of DRAM is a viable alternative, which is exactly why it is the specific product the whole shortage is about.

Why HBM specifically comes down to what a GPU does when a language model answers a prompt. The model generates its response one token at a time, and to produce each token it must read its entire set of model parameters out of memory. This can be hundreds of gigabytes for a large model and well over a terabyte for the biggest, not including the arithmetic that needs to be done to generate responses. So the chip spends most of its time waiting on memory rather than calculating, a limit engineers call the memory wall, where adding raw compute changes nothing and only faster, larger memory raises output. The parameters are only half of it: the context the model works with, the prompt, any retrieved documents, and the conversation so far, also lives in HBM, held as the KV cache (key-value cache) that stores everything the model has already read so it can reuse it rather than recompute it for each new token. That cache is what keeps the model coherent across a long prompt, and it grows in direct proportion to the length of the context, until at long context lengths it can exceed the model's own parameters. So, between the weights it must stream and the context it must hold, every request leans on high-bandwidth memory. Furthermore, the more context you feed a model, the more of it each request consumes.  

Why This Squeeze Is Structural, Not Cyclical

Memory has always moved in cycles, but this shortage is not a normal turn of one. It is the worst supply crunch the industry has seen in fifteen years, and it is structural, a product of how this particular kind of memory is made and of who is able to make it. Four forces are holding it in place, and their reach now extends well beyond the data center.

The first is that the memory AI depends on is exceptionally hard to manufacture. An HBM stack is not a single chip but eight or more DRAM dies bonded on top of one another with thousands of connections aligned across every layer, a process so demanding that yields are low and the finished part is, on the industry's own reckoning, as difficult and costly to produce as a leading-edge processor. The difficulty is not uniform across the producers: yielding the most advanced stacks reliably is currently the sharpest dividing line between suppliers, with one able to produce them at volume while another still struggles to. This is not a product whose output can be turned up at will.

The second is a straightforward contest for the capacity that does exist, and the AI buyers are winning it. Hyperscalers such as Google, Amazon, Microsoft and Meta are outbidding consumer device makers for the same production lines, and they are locking supply down for years rather than quarters: HBM output is effectively sold out for 2026, Broadcom has secured memory through 2028, and Micron alone is sitting on a contract book worth around $100 billion. Long-term commitments at premium prices mean the fabs serve the data center first and everyone else from what is left.

The third is that there is almost no one else to turn to. Essentially the entire world's supply of DRAM and HBM comes from just three companies, Samsung, SK Hynix and Micron, and that concentration is not a passing accident of the market. Building a competitive memory fab costs tens of billions of dollars and rests on decades of accumulated process know-how, a barrier so high that no smaller manufacturer and no new entrant can realistically cross it. As one memory-interface specialist put it, the hyperscalers have the resources to secure HBM while second-tier players are left making tradeoffs. When all three incumbents choose to prioritize AI, there is simply no fourth supplier to take up the slack.

The fourth is that the incumbents cannot expand on any short timeline. New fabrication plants cost tens of billions and take years to build; Micron has committed well over $150 billion across sites in Idaho, New York, Virginia and Taiwan, but meaningful output is not expected before 2027 at the earliest. Rather than extend cheaper legacy production to bridge the gap, the makers are doing the opposite, winding down older DDR4 lines and, in Micron's case, retiring its consumer Crucial brand at one of the most profitable moments in the company's history to concentrate on AI. And a second bottleneck sits just past the memory itself, in the advanced packaging that bonds HBM beside the GPU: that step is sold out as well, with lead times of roughly 52 to 78 weeks.

The result reaches far outside the server hall. On a PC, memory and storage have gone from about a sixth of the bill of materials to more than a third in a single year, by the account of HP's own finance chief. Nintendo raised the price of the Switch 2 to $499.99, citing memory costs directly.  

What This Means for the Way You Build

You do not control the price of high-bandwidth memory. You do control how much of it your stack wastes, and in a market like this, efficiency is the only lever that compounds in your favour. Here is where to focus.

• Make retrieval your default, not long context. Every token of context you load sits in the KV cache and consumes high-bandwidth memory on every step of a request, so a million-token prompt is a million-token cost repeated for each token generated. A retrieval-augmented design that fetches only the few relevant passage into a tight prompt does the same job at a fraction of the memory footprint, and for most enterprise cases it is also more accurate, because the model reasons over curated evidence rather than a haystack. This is the single highest-leverage decision on the list, and because it is architectural, it is far cheaper to get right early than to retrofit.

• Size the model to the task. Routing every request through the largest frontier model is, in a memory-constrained market, an expensive habit, because a bigger model streams more parameters from memory for every token it produces. Reserve the frontier model for the work that genuinely needs it and push routine, high-volume tasks to smaller or distilled models that carry a far lighter memory cost per call. Treat your models as a portfolio matched to workloads, not a single default that everything flows through.

• Treat efficiency techniques as a standing discipline, not a one-off. Quantisation, which runs a model at lower numerical precision, can cut the memory a request consumes by 30% to 50% with little quality loss, and prompt caching and request batching extract more work from every gigabyte of bandwidth. These stopped being niche optimisations the moment memory became the constraint. They are cost controls, and they belong on the roadmap of anyone serving AI at volume.

• Put memory-driven inflation into every budget, not just the AI one. The device fleet, the storage estate, and the hardware refresh cycle are all quietly more expensive now for reasons that have nothing to do with your AI roadmap. Fold that into planning before it surprises the finance team, because this is the effect most enterprises will feel and fail to attribute.

• Reset the build-versus-buy line. During a shortage, buying managed capacity usually beats building your own cluster, because the provider absorbs the procurement war you would otherwise fight. The exception is where data residency or sovereignty forces the workload to stay in your own jurisdiction, a live constraint across this region, and there the decision is about control rather than cost. Either way, treat inference capacity as a planned, contracted purchase rather than a spot buy, the same discipline the chip buyers themselves have now adopted.

The teams that come out ahead over the next two years will not be the ones that cornered the most compute. They will be the ones that learned to do more with every gigabyte. The scarce resource has moved, and the cost of getting that wrong is rising. Efficiency is now the strategy.