Back in October 2018, a bombshell rocked the tech industry when Bloomberg reported that some motherboards made by Supermicro had malicious components on them that were used to spy or interfere with the operation of the board, and that these motherboards were found on servers used by Amazon and Apple. We covered the event, looking at how it could work if it were true. Now seven months have passed, and it’s time to look at how things shook out.
No Evidence Yet, But Plenty Of Theories
Immediately after the news was reported, everybody tried to get their hands on images or physical servers that were compromised in an effort to verify the claims, and nobody has succeeded in finding anything independently. In addition, Apple and Amazon filed unequivocal denial statements, and Supermicro filed a letter with the SEC telling customers it was confident that this story was false. Then they hired a third party auditor, who found no evidence of any tampering. If there was something there, either nobody has found it after 6 months (highly unlikely), or there’s a conspiracy of gargantuan scale (even more unlikely).
In a presentation at Chaos Communication Congress Trammell Hudson did a thorough investigation on this topic and the talk is very well done and pulls together research from other people as well as his own. While he agrees that Supermicro’s manufacturing process probably wasn’t compromised, he does point out that government agencies have been known to intercept freight and carefully modify the hardware before resealing it and sending it on its way. Whether that’s happening in China on the way out or the U.S. on the way in is unknown. He also mentioned the possibility of the supply chain being compromised before manufacturing and fake chips being sent to the manufacturer.
He managed to succeed in hacking the BMC with what was essentially a single component that could replace a resistor on the board, demonstrating with his proof of concept that it was plausible to do what Bloomberg’s reporting claimed was being done.
Fallout From The Blast
The manufacturer initially took a huge hit to their stock value, but as of April it had returned to the level it was at before the news. In their quarterly earnings report there was definitely a decline in sales in the last three months of 2018 (from $952 million the previous quarter down to $915 million), with estimates of a similar drop in the first three months of 2019 (the numbers aren’t out yet). In other words, this hurt Supermicro on the order of tens of millions of dollars in lost revenue, and possibly more in damage to the brand, but it was not a fatal blow.
They are just breaking ground on a new 800,000 square foot, $65 million plant in Taiwan and are expanding their Silicon Valley headquarters. This is at least in part because some clients have asked Supermicro (and other manufacturers) to move out of China because of security concerns. It may also be because of tariffs that have made China production more expensive. The shift away from China had already started before October, but it accelerated afterward.
The effects on Bloomberg were essentially nonexistent. Maybe they’ve lost a little credibility, though it’s hard to tell. In the immediate period after the publication they stood by their article and the research they did. However, they have not published any more information to back up their claim, nor have they published a retraction. If anything, Bloomberg has doubled down.
A few days after the report about Supermicro, they published another separate accusation, this time claiming that the motherboards had Ethernet connectors with malicious hardware inside of them. However, shortly after that, the person quoted in that article said he was misrepresented and that he wasn’t trying to single out Supermicro but instead say that the problem was industry-wide.
The two authors of the reports, Jordan Robertson and Michael Riley, haven’t published anything for Bloomberg since. Maybe they’re working on their next piece, or getting to the bottom of this one.
Scrutiny Goes Beyond Supermicro
Supermicro hasn’t been the only one under scrutiny lately. Huawei has also been under fire for having hidden backdoors in their communications equipment. This reporting, also by Bloomberg, is different because this time there’s corroboration. In the wake of this, Huawei is being banned in a few countries, and it’s starting to hurt the company. Many manufacturers are leaving China and moving to other countries, as the threat of China hacking, the increasing costs of labor, quality concerns, and rising tariffs make moving more and more appealing. Supermicro and Huawei are just illustrative examples of the trend.
On the other hand, Cisco just released an announcement about a hidden backdoor in a server (and a patch to fix it), so maybe Huawei just had a firmware bug and didn’t handle it well.
Many people have since agreed that the theory behind the kind of hardware hacking claimed by Bloomberg is sound, though it’s extremely challenging to pull off. Supply chain management, vendor management, and managing certifications and integrity of vendors internationally for complex components is a nightmare, and it wouldn’t be unheard of for a vendor to slip in some components of questionable provenance.
It wouldn’t be easy, though, with so many test and verification steps performed by so many organizations. Adding a new component would be nearly impossible since it would require numerous changes (like alterations to gerber files, the pick and place programs, the automated optical inspection, and the in-circuit test), but replacing an existing one with a similar but malicious component would be harder to detect. We’ve seen lots of instances where fake components make it into the supply chain without the knowledge of the manufacturer or the customer, so it’s a little more believable that this is the vector.
Moving out of China doesn’t completely mitigate the risk, though, as many components are only manufactured in China. Companies are getting more vigilant about monitoring their supply chain and eliminating the possibility of this security problem.
In Conclusion, No Conclusion
Something is up and the story isn’t over. We still haven’t seen the smoking gun Bloomberg claimed with Supermicro, but they haven’t retracted, either. Supermicro is on the mend after all this, and they are among many in an exodus from the security risk of manufacturing in China. The story with Huawei is still developing, and it’s very difficult to tell if they are villain, victim, or somewhere in between. In the meantime, we should be boning up on our secure communication skills, our firewall rules, and monitoring our supply chains just in case a story turns out to be true.