It’s definitely a sign of the times when repair shops get handed sticks of registered ECC DDR5 RAM by businesses to try and fix, rather than just tossing the defective ones and slotting in a fresh bunch. This is how [northwestrepair] ended up with two of these ECC RDIMMs, with a current market valuation of a rather nice budget car.
The 128 GB stick was first on the bench, with the fault being that it would fail DDR5 training on boot. This suggests that one or more of the DRAM BGA packages had developed a cracked solder joint, which is something that you could conceivably fix with more diagnosing and testing for a single IC reballing.
In this case doing a reflow cycle was enough to bring the RAM back to life, with the DDR5 training now passing without issues in a test system.
For the 256 GB stick things wouldn’t be so easy, as it seemed completely dead. This suggested a dead PMIC, as this is the component on DDR5 RAM responsible for ensuring that the DRAM ICs even power on. The SPD ROM was another option, so that got swapped first, which does require reprogramming the RDIMM. For this the earlier SPD dump image was reflashed, but swapping this chip made no difference.
After some additional reflowing of a few suspect ICs, still nothing happened. Following this a lot of reballing and other troubleshooting steps followed, only for the problem to be one shorted MLCC and a broken PMIC, potentially caused by whoever dropped said RDIMM onto the server room floor.
Cracked MLCCs do appear to be a very common cause of failures, as they’re often used as filter capacitors, same as for this RDIMM. Checking for shorts across a number of such MLCCs can be a good way to shorten a diagnostic cycle. Since this RDIMM was dropped, reballing every IC probably was still a good move as this may have cracked some solder balls.
Up next repairing hard drives which should be a book into itself.
The shortage has now lasted a significant fraction of the length of WWII, when production was ramped up such that nearly all industries went 2x, and some specific crucial industries did 20x-35x.
I have heard the explanations of the bottlenecks and why we simply can’t do this, but it doesn’t really add up imo. It’s all just engineering problems, and there’s so much money to be made. What’s the deal? Is this a roundabout method to keep AI development under the speed limit? Maybe not intentional, but once people saw it functioning that way they decide not to build more of the fancy optical systems needed to increase capacity? That all still sounds a bit to conspiratorial.
Component complexity is the killer here. Say you need more rivets. You can mine more iron, open more smelters to process it, and more forges to make the rivet. You’ll need miners, picks, furnaces and dies.
If you want to make RAM, you need to mine more silicon, refined it, grow more crystals (this takes months), facilities for cutting and making wafers, patterning tools, packaging tools…
You need to make more tools to make more tools to make tools to make your product.
You forgot about helium used in operating wafer machines. Since the unrest in Ukraine started in 2022 helium avability went belly up. I had my little TIG welding workshop but had to cease operations in 2024 because price of helium gas was just too high.
Sadly most of factories which manufactured purified helium were located in Donbas and are now ruined. This was old but reliable 1970s technology maintained by engineers and technicans, but they are all now drafted or died in war so there’s no one to recreate old machines and start making new helium.
I did absolutely forget about that, good tip, thanks
There has also been “comparatively” zero effort to capture helium from underground drilling in North America since WW1 when they hoarded it, thinking that Zeppelins would maintain relevance.
We stopped collecting it, we stopped storing it, and the US started selling it off for next to nothing because… keeping it in a hole in the ground wasn’t convenient, I guess?
We got cheap party balloons but we will see if it comes at the cost of MRI machines.
https://en.wikipedia.org/wiki/National_Helium_Reserve
https://www.nbcnews.com/health/health-news/us-just-sold-helium-stockpile-s-medical-world-worried-rcna134785
81 million cubic meters of Helium were extracted from natural gas in the USA in 2025. 119 million cubic meters were extracted in the rest of the world. https://pubs.usgs.gov/periodicals/mcs2026/mcs2026-helium.pdf
Also, the lng plant that sits on the strait of Hormuz was one of the world’s largest suppliers of helium. I hope this will get us to stop using it in balloons.
I get that, and it makes sense… I just think that at some point it would still get brute-forced.
We also have tons more automation now than we had then. It’s more difficult than rivets, but we had to gear up RAM production to get where we are in the first place, and we should be able to do it again. I know that it has ramped up a little bit, just not enough to actually meet demand in any meaningful way that would prevent two sticks of RAM from costing a pretty serviceable Toyota Corolla.
Which we don’t actually need in most cases of computing. And could work around with a simpler architecture, in the others.
edit: what i wanted to say is, this complexity is the side-effect of a market centered around cloud-computing (the hardware being made for their demands) and the PC business getting the scraps on the side.
I’m convinced that a maybe 50 nanometer fab (each industrial nation has a few of them around) with a simple and robust computing architecture in the MHz range (and respective software infrastructure), with a guaranteed operating time of 10 years, could be a progress over todays GHz and Gigabyte systems. Sure, it’s alternate-universe in scope, compared to how professional software dev is still in the chopchopchop-libraries-and-gluecode phase, but i think the various microcontroller-OS are a step in the right direction.
The simplest (if not completely accurate and completely unsourced) answer is that DRAM manufactures don’t want to get bitten by a fall in demand if the AI bubble pops. It takes a LONG time to bring a DRAM fab online, years. When smartphones suddenly needed a lot of DRAM, companies rushed to open new fabs. By the time these new fabs opened the massive demand for DRAM had dried up and the companies had to foot the bill. This time around they decided to just let the price spike and keep their existing expansion timelines. That gives them a lot of immediate profit and shelters them from losses in case the new fabs end up being unnecessary. They were already spinning up new fabs but not to the scale that current demand requires.
…That’s a very sensible point actually, why didn’t I think of that? Thanks
Plus you would need a lot of extra volume at reasonable prices to make more profit then the 90%+ margin they are making now.
Exactly. If/when the bubble pops, they just go back to the way it was before. Where they sell to pc builders and hardware manufacturers. It is a smart way to shield themselves from the bubble, but sucks for us.
In most extreme fabs are anchored directly into solid bedrock to achieve absolute stillness (via tens of thousands of deep shafts, filled with concrete). The mass helps to attenuate vibrations. Then the cleanrooms are floated above this foundation to further dampen vibrations. There is an insane amount of analysis and calibration that can last years involved. And not all locations around the world are suitable.
Yeah, I have heard the difficulty of production described and I believe it, I just also know that they built those fabs in the first place, and there’s definitely still suitable spots to build more if they needed to. Even if it’s extremely difficult, there’s just no way that we are incapable of making more of this commodity, and there’s a huge profit incentive to do so, so maybe there’s some non-engineering reason I’m missing. Some people here have brought up a few reasonable options
I’ll expose my ignorance – assuming this is needed for 2 nanometre chips, what if you “settled” for five nanometre designs?
Or ten, even twenty nanometre?
Well obviously the chips get bigger, and RAM per chip goes down, but your fab costs / requirements should also drop. If the engraving beam suddenly has (for example) a 3nm wobble tolerance for 10nm designs, there’s got to be a cash value saving associated.
Paradox of chip design is the smaller the node, the harder it is to switch it without electrons flying through silicon. With 130 nm Northwood chips it was rather easy to overclock, now it’s not so easy because clock speed is more restricted. Capacitive effects also play a role in loss so the more transistors switch in unit of surface the more capacitance gets shovvled about and the higher the current. At some point you must decide if you want more cores, more speed or more transistors because with constant power the chip will melt.
Energy consumption and speed:
Bigger chips waste more energy and this transforms in heat. On the one hand we’ll never have enough energy and on the other hand all that heat concentrated in such a small footprint burns those things up.
The bigger chips also have longer communication paths making them slower.
Why would they want to sink billions into new fabs just so they can sell their products for less money? From their point of view, everything is fking awesome right now.
If this made any sense, then why would any industrial producer ever develop economies of scale? They almost unanimously do choose to scale up and usually end up making astronomically more money afterwards. I think you need to spend more time on that theory.
The exception would be if demand immediately collapses afterwards and they’re left holding the bag, as G details above.. But I kind of don’t really buy that RAM demand is going away any time soon. Maybe it’ll level off or even reduce, not sure.
You are confusing Adam Smith Capitalism with Jack Welch Capitalism. The former is ‘make good products at a fair price’, the latter is ‘who cares if it makes any sense as long as number go up’.
Fair dibs
you’re missing the amount of capital involved, and barriers to entry.
Often business will make more of something (instead of just putting prices up) because if they don’t someone else will.
However, if that someone else will need an large amount of capital, and have significant issues making it, you don’t have to worry about it as much and a business can simply put the price up – it does happen quite often, though this is an extreme case..
As one of the posts above says, why should they spend billions to make more capacity to lower price? Instead they can just act put the prices up ie act like any monopoly does..
That’s the sort of thinking that Henry Ford spotted. He realised there was a whole untapped market for an effective car rather than a luxury car.
my understanding of the shortage is that all of this stuff being produced in substitution for what consumers would buy is all accumulating unused in warehouses, since the server farms aren’t built and even if they were, there would be no power for them. at some point the dumb money (which is all that is causing this) will realize there is no business model that works for LLMs and the whole thing will collapse, at which point consumers will have to figure out what to do with the high bandwidth memory that is being warehoused. hopefully there will be special motherboards soon thereafter
Do you really think there’s no business model that works for LLMs? I was under the impression that was a bit of an out-of-date talking point… I cannot find a single position in my company or in any of my client’s companies that does not require use of LLMs in some capacity. Sometimes very heavy use. I’m trying to say if it’s a good or bad thing, time will tell… but most of the people I know have a subscription to buy tokens for their real job.
Or is it that token sales don’t break even/the companies are still in the red? I mean Uber has operated that way until very recently, so it’s not strictly disqualifying…
This. Everybody is using it because the companies providing it are selling it under cost.
Will people keep using it at a price that the providers can make money and pay back all the losses they’ve had to take? No idea.
No job “requires” using an LLM. we’ve been doing the work without them for centuries.
Taking you at your word, that all of the jobs at all of your clients benefit from using LLMs, I’d suggest that you have sample bias and your industry sector is not representative of the entire job market. LLMs are an overhyped tool that do a handful of tasks really well but do not solve every problem in every industry. In many cases LLMs are counter productive due to the overhead of needing to check their work and already be an expert in the field in order to spot hallucinations and spurious “reasoning”.
The whole silicon valley plan is to operate at a loss surviving off of venture capital funding until you’ve cornered the market enough and purchased enough of your competitors that you can jack up your prices and progress to the next stage of enshitification. When founders and funders talk about “disrupting the market” they mean “reorganize the market with my company at the top doing the exploitation” not “improve the market to the benefit of everyone.” It’s an inherently extractive view of the economy
also — the enshittification playbook cannot work on LLMs. they are interchangeable (fungible) in a way that social networks or entrenched software suites are not, so that the moment the squeeze is applied, people switch to a good-enough alternative (of which there are many). this happens already – i am in a conversation with ChatGPT (which, like 95% of people who use it, i do not pay for — because f*ck those a$$holes!) when it tells me that I have to pay in order to continue our conversation. i pick up where i left off with Google Gemini and it’s not even a minor inconvenience.
The dumb money thinks LLM companies are just like Uber, Amazon, or Facebook — that being a first mover or network effects will make running up huge debts payoff in the end. But none of the user an LLM company gets can be relied on to stay, since there is nothing one of these companies provides that cannot be easily replaced by another. So running up huge debts and building massive data centers gets you nothing that China can’t replace for much cheaper. Another thing about LLM users is that they do not scale anything like Facebook users. Adding Facebook (or Uber) users costs nothing, because each requires tiny amounts of server support. But every LLM user is requiring a significant chunk of compute be available.
Advanced silicon is the most sophisticated supply chain on the planet. For the West, it’s not just about AI, it’s about hoarding as much of that supply as possible before the Era of Globalized Logistics comes to an end and buy more time for the CHIPs Act to fulfil its purpose.
What no one seems to have mentioned yet is that the memory makers have already been convicted of operating as a cartel and colluding to keep prices high in the past.
There are already lawsuits working their way through the system suggesting they are doing it again. Part of the evidence is that they seem to be purposely making less of the more profitable DDR5 chips for more HBM which is forcing the prices to stay elevated.
You should check out Gamers Nexus and their videos on the memory cartel for more info.
But yeah, there is reason to believe that the reason things haven’t ramped is because all the companies are in agreement so that no one of them suddenly ramps production, swipes up marketshare and makes a boatload of cash. Instead they are working together to keep supply low so they can sell for insane prices with no extra cost.
There is more info and discussion about some interesting events around Chinese memory makers competing. you should really check out Gamers Nexus, they are doing a lot of big investigative work lately.
RAM has been very cheap and abundant for years. The companies don’t build out, because this way they can ask insane prices an run massive profits. So why would they build. Who is going to use the extra expensive capacity once the bubble collapses or all the data centres are build? Or is someone found a way to reduce AI memory use…..
The reality is, there is no profit in expanding production capacity for existing player. The new supply usually has to come from new player (from china and such). That see a way to start and take a piece of the pie. Player normally nobody would buy from, but desperate people wil.
I saw a good post on Reddit where someone pointed out that it takes ~5 years and ~$25bn to build that chip fab which as you say is entirely possible.
However – if you have $25bn you could invest it elsewhere and make ~9% return every year reasonably safely, so your chip fab project has to be fairly confident in doing much better than that every year for the next 10+ years which is by no means guaranteed – by the time you build it the crisis could be over, prices could collapse, a new technique could arrive and render your fab obsolete, etc. etc.
I don’t remember who said this, I think it was an executive at Samsung. They said that they don’t expect the spike in demand for RAM to last very long. If they invest in more capacity, they need sustained demand for the next 10+ years to recoup their investment. So it doesn’t make economic sense for them to build new fabs just for RAM.
Bad PCB design, inner layers reach end of the board!
Dear Hackaday, please define your abbreviations, especially when they aren’t common. Half of this article is alphabet soup.
None of the abbreviations in this article are close to what I would consider uncommon (especially considering the audience here on hackaday), though I agree it would have been better if they did define them in their first use.
I had to lookup MLCC, as it wasn’t defined or described. I let PMIC slide, because it’s purpose was laid out. As long as you’re giving me an idea what it is/does, I’m OK with it. Toss one out there willy nilly? Now we’ve got a problem.
If you drive an automobile in a busy city, you know the annoyance of red-light delay caused by the “Slinky” or accordion effect. http://en.wikipedia.org/wiki/Accordion_effect. Silicon chip manufacturing bandwidth suffers from the same Slinky effect with each major product run requiring new setups and certifications to ensure high yield.
Using TSMC as an example:
Contrary to popular belief, existing wafer facilities are funded 75% to 90% by internal cash financing and not external cash sources. TSMC operates exclusively as a pure-play foundry, meaning virtually 100% of its revenue and profit stems from fabricating silicon wafers in its fabs for external designers. TSMS can do this because their NET PROFIT is 55.6%, turning more than half of total wafer sales directly into bottom-line profit.
DRAM buying does not follow a traditional stock market “futures” approach; rather DRAM is constantly evolving and the demand market is unevenly divided in the various DRAM technologies, speed, and compatibilities. Manufacturers prefer direct pricing power through corporate contracts rather than leaving price discovery to an open speculative exchange. The market is shifting slowly with offerings from ETF traders and Crypto Perpetual Futures which are de-centralized platforms and have introduced huge volutility as the online exchanges are world wide and 7×24.
http://finance.yahoo.com/technology/articles/global-dram-market-report-2026-110100507.html
Finally, consumer PCs are DDR-based for the most part while datacenters demand HBM/ECC modules. Any datacenter surplus inventory is highly unlikely to be utilized in consumer products. Thus, as fabs are retooled, the Slinky-effect again causes volatility.
For better or worst, capitalism is a 2-edged sword.
Of the two, the reflowed 128 GB stick is the one to watch once it’s back in a server. Reflow remakes the existing joints in place, but the solder underneath is the same fatigued solder, which is why repair benches tend to quote reflow-only fixes as the less durable option and reballing as the lasting one. If that joint opens up again, the first sign in a server is more likely to be corrected ECC errors on that one module than another training failure.
Linux makes that easy to watch per module. EDAC keeps dimm_ce_count and dimm_ue_count for every DIMM under /sys/devices/system/edac/mc/, and dmidecode -t 17 maps each slot’s locator to the module’s serial and part number. In a box full of identical 128 GB RDIMMs, the serial is the only way to tell which one went back through rework.
If the corrected count on that slot starts climbing while the others stay flat, swap it at the next maintenance window instead of waiting for an uncorrectable error to take the host down. The kernel’s own RAS notes say proactive replacement of modules that show corrected errors can reduce the chance of exactly that.