The AI memory squeeze: what Samsung's and SK hynix's record quarters mean for your inference architecture

What happened

On 29 October, Samsung Electronics confirmed record quarterly operating profit of about 107.4 trillion won, almost all of it from its chip business, driven by high-bandwidth memory (HBM) for AI servers¹. A day earlier, SK hynix reported its own record quarter, said HBM4 had entered full-scale mass production, and described memory prices rising across HBM, server DRAM and enterprise SSDs².

TrendForce's latest forecast, from 30 September, expected conventional DRAM contract prices to keep rising in the fourth quarter, with server DRAM still undersupplied³. In the same week, Korea's FuriosaAI raised $620 million to scale its RNGD inference chip, pitched on efficiency per watt⁴.

The common thread: memory is one of the scarcest, fastest-rising costs in AI infrastructure. That matters whether you rent GPUs in the cloud or buy your own.

Why inference is a memory problem

Serving a large language model needs memory for two things: the model's weights, which are fixed, and the key-value (KV) cache, which grows with every active conversation and every token of context. The KV cache is what fills up under load. The vLLM team's PagedAttention paper showed that much of this memory was wasted by naive allocation, and that managing it like virtual memory pages let one GPU serve far more requests at once⁵.

Agents make this worse. A long-running agent carries a large, growing context, and many agents in parallel hold many caches at once. If your plans assume agent workloads will grow, assume memory pressure will grow faster than request counts.

What to change in the architecture

Route with memory in mind. A standard load balancer spreads requests evenly and ignores how full each replica's cache is. The Kubernetes Gateway API Inference Extension routes using model-server metrics; in its published benchmark, throughput was similar to a standard Service but tail latency was much lower at high load, which the authors attribute to fewer collisions as GPU memory nears saturation⁶.

Cache shared prefixes. If every request starts with the same long system prompt or document set, reuse its cache instead of recomputing and storing it per request. See prompt caching for cost and latency.

Right-size and quantise. Not every step needs the largest model. Model routing sends simple steps to smaller models, and quantisation cuts weight memory, as long as your evaluation set shows quality holds.

Cap context deliberately. Long context is convenient but expensive in memory. Retrieval and summarisation often beat sending everything; our long context versus RAG note covers the trade-off.

Treat accelerators as a pool you can change. Kubernetes exposes accelerators to workloads through device plugins and, more recently, Dynamic Resource Allocation. Keeping serving on that standard layer, behind a portable engine such as vLLM, is what lets you test a new accelerator, such as the ones FuriosaAI and others now sell, without rewriting your platform. See DRA for GPU scheduling.

What to change in buying

• Quote early, and quote again. If you plan on-premises inference, as in private LLMs on-premises, memory-heavy servers are rising in price. Lock quotes where you can, and size purchases to the models you will actually run.
• Compare cost per served request, not per GPU. A card with more memory may serve more concurrent users and cost less per request, even at a higher list price.
• Benchmark alternatives on your own traffic. Vendor figures, including FuriosaAI's, are the vendor's own⁴. Replay real prompts and measure.
• Ask cloud providers about memory, not just GPU type. The same GPU name can come with different memory sizes and very different serving capacity.

Who should act, by region

Teams building sovereign or in-country AI capacity feel this most, because they cannot always fall back on a hyperscaler's pooled supply. That includes public-sector and regulated buyers in the GCC, India, the EU and Southeast Asia. For cloud-first teams in the United States and elsewhere, the impact arrives through instance pricing and capacity, which is covered in planning around the AI capacity crunch.

A 60-day plan

CTO take

Record memory profits are a signal about your costs. As long as AI demand outruns memory supply, the cheapest capacity is the capacity you stop wasting. Most inference estates I review have room to serve noticeably more traffic on the same hardware through better caching, routing and model sizing. Do that work before signing the next hardware order, and keep the platform portable enough to take advantage when alternatives get cheaper.

For the wider picture, see today's tech news brief.

How I can help

I review inference platforms for cost per request, latency and portability, and help teams plan hardware and cloud commitments. I am based in India, the United States and the UAE, working globally. See AI architecture or analyse your challenge and bring the result to a call.