Back in October 2018, a bombshell rocked the tech industry when Bloomberg reported that some motherboards made by Supermicro had malicious components on them that were used to spy or interfere with the operation of the board, and that these motherboards were found on servers used by Amazon and Apple. We covered the event, looking at how it could work if it were true. Now seven months have passed, and it’s time to look at how things shook out.
No Evidence Yet, But Plenty Of Theories
Immediately after the news was reported, everybody tried to get their hands on images or physical servers that were compromised in an effort to verify the claims, and nobody has succeeded in finding anything independently. In addition, Apple and Amazon filed unequivocal denial statements, and Supermicro filed a letter with the SEC telling customers it was confident that this story was false. Then they hired a third party auditor, who found no evidence of any tampering. If there was something there, either nobody has found it after 6 months (highly unlikely), or there’s a conspiracy of gargantuan scale (even more unlikely).
In a presentation at Chaos Communication Congress Trammell Hudson did a thorough investigation on this topic and the talk is very well done and pulls together research from other people as well as his own. While he agrees that Supermicro’s manufacturing process probably wasn’t compromised, he does point out that government agencies have been known to intercept freight and carefully modify the hardware before resealing it and sending it on its way. Whether that’s happening in China on the way out or the U.S. on the way in is unknown. He also mentioned the possibility of the supply chain being compromised before manufacturing and fake chips being sent to the manufacturer.
He managed to succeed in hacking the BMC with what was essentially a single component that could replace a resistor on the board, demonstrating with his proof of concept that it was plausible to do what Bloomberg’s reporting claimed was being done.
Fallout From The Blast
The manufacturer initially took a huge hit to their stock value, but as of April it had returned to the level it was at before the news. In their quarterly earnings report there was definitely a decline in sales in the last three months of 2018 (from $952 million the previous quarter down to $915 million), with estimates of a similar drop in the first three months of 2019 (the numbers aren’t out yet). In other words, this hurt Supermicro on the order of tens of millions of dollars in lost revenue, and possibly more in damage to the brand, but it was not a fatal blow.
They are just breaking ground on a new 800,000 square foot, $65 million plant in Taiwan and are expanding their Silicon Valley headquarters. This is at least in part because some clients have asked Supermicro (and other manufacturers) to move out of China because of security concerns. It may also be because of tariffs that have made China production more expensive. The shift away from China had already started before October, but it accelerated afterward.
The effects on Bloomberg were essentially nonexistent. Maybe they’ve lost a little credibility, though it’s hard to tell. In the immediate period after the publication they stood by their article and the research they did. However, they have not published any more information to back up their claim, nor have they published a retraction. If anything, Bloomberg has doubled down.
A few days after the report about Supermicro, they published another separate accusation, this time claiming that the motherboards had Ethernet connectors with malicious hardware inside of them. However, shortly after that, the person quoted in that article said he was misrepresented and that he wasn’t trying to single out Supermicro but instead say that the problem was industry-wide.
The two authors of the reports, Jordan Robertson and Michael Riley, haven’t published anything for Bloomberg since. Maybe they’re working on their next piece, or getting to the bottom of this one.
Scrutiny Goes Beyond Supermicro
Supermicro hasn’t been the only one under scrutiny lately. Huawei has also been under fire for having hidden backdoors in their communications equipment. This reporting, also by Bloomberg, is different because this time there’s corroboration. In the wake of this, Huawei is being banned in a few countries, and it’s starting to hurt the company. Many manufacturers are leaving China and moving to other countries, as the threat of China hacking, the increasing costs of labor, quality concerns, and rising tariffs make moving more and more appealing. Supermicro and Huawei are just illustrative examples of the trend.
On the other hand, Cisco just released an announcement about a hidden backdoor in a server (and a patch to fix it), so maybe Huawei just had a firmware bug and didn’t handle it well.
Many people have since agreed that the theory behind the kind of hardware hacking claimed by Bloomberg is sound, though it’s extremely challenging to pull off. Supply chain management, vendor management, and managing certifications and integrity of vendors internationally for complex components is a nightmare, and it wouldn’t be unheard of for a vendor to slip in some components of questionable provenance.
It wouldn’t be easy, though, with so many test and verification steps performed by so many organizations. Adding a new component would be nearly impossible since it would require numerous changes (like alterations to gerber files, the pick and place programs, the automated optical inspection, and the in-circuit test), but replacing an existing one with a similar but malicious component would be harder to detect. We’ve seen lots of instances where fake components make it into the supply chain without the knowledge of the manufacturer or the customer, so it’s a little more believable that this is the vector.
Moving out of China doesn’t completely mitigate the risk, though, as many components are only manufactured in China. Companies are getting more vigilant about monitoring their supply chain and eliminating the possibility of this security problem.
In Conclusion, No Conclusion
Something is up and the story isn’t over. We still haven’t seen the smoking gun Bloomberg claimed with Supermicro, but they haven’t retracted, either. Supermicro is on the mend after all this, and they are among many in an exodus from the security risk of manufacturing in China. The story with Huawei is still developing, and it’s very difficult to tell if they are villain, victim, or somewhere in between. In the meantime, we should be boning up on our secure communication skills, our firewall rules, and monitoring our supply chains just in case a story turns out to be true.
“Many manufacturers are leaving China and moving to other countries, as the threat of China hacking, the increasing costs of labor, quality concerns, and rising tariffs make moving more and more appealing.”
Yup, to countries that don’t spy on their citizens. ;-)
Meanwhile, in Room 641A…
I did not catch the reference. For those who also missed this: https://en.wikipedia.org/wiki/Room_641A
As Spock might say, “fascinating” (and a bit 1984ish scary.)
Yeah, not much talk of Mark Klien’s papers, he literally showed the visio diagrams for domestiic surveilance of internet fibre splitters in the US.
If i were running Huawei or Supermicro or any of these Chinese tech giants trying to sell to the west, I’d go full FOSS in design, rub a Stallmanesqe indepandant audit coupled with a full hardware/firmware/tool chain-repo on github for products critical to public infrastructure or with over a million individual deployments.
Fact is that Western tech companies develop back door laiden systems as much as anyone. Companies like Cisco and IBM sell deep pack inspection and surviellance tech to oppressive regimes round the world. no validation process, no audit trail, just trust us.
Systems that affect the communications and privacy of millions of people need to be peer review audited independantly like financial records, medical procedures and civil engineering are. We’re gonna get burned over and over until they are.
Which countries are those? Asking for a friend.
I don’t think anyone believes there is a country that does not spy on its people to some degree and also does not tamper/sabotage device production in their factories. It really is more of a situation of, is their spying at a tolerable invasiveness and is the risk of counterfeit parts or illicit changes to design at a tolerable risk. China not only is known for being incredibly invasive in how it monitors its citizens and outsiders, but the risk of counterfeit parts and design changes that result in hidden and/or unexpected behavior or failures is now at a risk that can no longer be overlooked.
If you judge a civilization by how they treat their prisoners, then you can judge a country by how they treat their citizens.
Non consensual monitoring is the norm within the borders of China. Why does anyone think there would be the slightest hesitation outside their borders?
With access to the hardware and design documents you can choose your path for intrusion.
@RandyKC: In a pragmatic point of view, I judge a country in how they behave in the world stage. US as part of the Five Eyes violates the privacy of everybody.
US sphere of influence affects me vs China not so much.
e.g. Evo Morales grounding incident – Morales’s Dassault Falcon 900, carrying him back to Bolivia from Russia, took off from Vnukovo Airport, but was rerouted to Austria when France, Spain, and Italy[2] reportedly denied access to their airspace, allegedly due to suspicions that Snowden was on board.
They weaponize their financial power these days and can unilateral initiate trade wars that affects nations for friend and foes alike.
Huh? GOOGLE knows more about everybody than the NSA.
Because how hard can it be… It’s all just ones and zeros after all… Anyone can declare a new country, acceptance is the hard part. Be a country that doesn’t spy on it’s own citizens and you’ll get a pretty big following.
https://m.wikihow.com/Start-Your-Own-Country
No, you won’t, you’ll be dismissed as a crank who probably smells of wee and stale cannabis smoke.
It is possible that the investigation that actually found evidence of tampering was classified, depending on where any of these mythical, compromised servers ended up.
That would make the Bloomberg story even more of a lie. Remember that this manipulation was allegedly found before the hardware was deployed and analysed by e.g. Apple. Why would that be classified? Why would Apple make statements that make them legally exposed when the correct answer to any classified data is “no comments”?
Are you implying Bloomberg would lie?
Exactly, Why in the world would Amazon or Apple admit to having China Spy ware on their servers. I think there is a huge coverup!
William of Ockham would say hacking the motherboard in this fashion is an unnecessarily and overly complex way of hacking a server’s security and thus likely untrue at a cleanly shaven face value.
Yes. The razor would also shred the claim of a compromised server being able to circumvent a competently designed secure network by magic.
This is where the story falls apart for me.
Yes, this is technically possible, but the story discusses how these were detected before production use, so they would be in a (theoretically) even more scrutinised network removed from customer data.
So they found something, that was unable to do anything, and then what. Just said they did and destroyed all evidence?
Reporting is one thing, but this story bothered me because it was written like a bad tv episode.
Enough details to say something is possible, but nothing happened, and then the episode ends without answering any questions at all.
The razor also shreds the need for extra parts – there are hardware backdoors even beneath the system management engine in Intel (and whatever they call it) and AMD cpus. You can get to ring -2! Right down in chipset firmware, persistent and “whatever you want”.
https://www.youtube.com/results?search_query=defcon+god+mode
That’s assuming any of it is other than Bloomberg sensationalism. He should go back to worrying about soft drink cup size. His news outfit is openly biased towards his views, his favorite companies and so on.
Yup. Little Billy Ockham would have pulled off a software/firmware attack on the BMC.
As for how this got reported, it really looks like the reporters were working on this “story” for more than a year, and disregarding everything negative that their sources said to them. We’ve talked to two of their sources who said “yeah, possible, but there’s an easier way so I doubt it”. The Bloomberg reporters kept the “it’s possible” angle.
Ockham’s razor here points to simple overzealous journalists chasing a “hot” lead, and refusing to listen to non-confirming interviews. Couple that with poor oversight by non-tech-savvy editors, and you get a Supermicro story. Sigh.
Journalism, like science, is also made of people.
It’s political. The timing is key.
This accusation was at the height of the US-China trade war.
The Bloomberg article itself is a weapon against Chinese economic interests and has already paid off for the US economic interests in this trade war.
Espionage and supply chain hacks are difficult, costly and carry a high risk of being caught.
It is MUCH easier to strike a political blow to an enemy with a simple lie that would be effective for months or years before it is found without merit.
Enough time has passed, and the Bloomberg folks came out with exactly _none_ of the proof that people wanted. The article was written on (multiple) anonymous sources — folks who didn’t want to go on the record, or imaginations of the author.
We’ve talked to some of the interviewees in person since then, and they have suggested that the reporter had an agenda and kept asking them leading questions.
The whole thing looks like a reporter with a pet idea (not borne out by facts) and some poor reviewing by Bloomberg editorial. I don’t know what corrective actions they’ve taken internally, but for me, they’ve lost a lot of credibility.
It’s all about political interests…
It has seemed just a maneuver to bring down Supermicro shares value.
Bloomberg is not a technical journal. Doubt they have the necessary technical compentence to analyse a history like this, or reflash a spi bios in a motherboard.
And all the people in the interviews mix too much the notions of possibility ( given the desire and enough money, almost anything is possible ) and probability of something being done/happen . Is it possible for a computer to die if I give it a good slap ? Yes. Is it probable ? Not so much.
If we are to go for the conspiracy theories, mine is that this was
1. to make supermicro prices fall and damage the company , for $MOTIVES
2. to conduce people to worry about this and overlook some other thing that was happening or being discovered at the time
Follow the money
This is a useless movie-fantasy platitude
25million shares traded in 48hrs in a stock that barely sees 100k trades a day.
Someone made a killing…
3. to make supermicro prices fall and buy them. From the chart it looks like you coul buy them for 12$ and they are now worth over 20$. Around 100% in around half year.
Knowing the story was going to be published you could short SuperMicro beforehand. So you could make money on the drop and by buying at the bottom and holding as the price recovered.
The SEC is pretty good at looking for these kinds of market manipulations. I haven’t heard anything yet.
This seems the most likely. Look at that graph…
In the world of camera phones everywhere and social media, I found it very unlikely that no one have ever caught the alleged modded chip at all. Writing such an article without even showing the actual picture is irresponsible journalism.
Bloomberg’s picture in their article is an illustration not the actual picture.
Trade war. That is the simplest theory. Get US companies to resist working in China.
It’s like Inception… gotta boil it down to a basic ‘idea’.
Bloomberg is a shit propagandas outlet. Week before they reported a major backdoor in Huawei core routers somewhere in Italy. Turned to be a common telnet service at :21 and lan access only :).
Hmmm, someone was shorting SuperMicro…
Bloomberg’s television systems use Supermicro hardware. I wonder if they’re aware of that, and if they’re worried by their own fear mongering.
“or there’s a conspiracy of gargantuan scale (even more unlikely)” unlikely, but fun!
It’s just politics. I wouldn’t think this is playing fair by their standards.
My company uses Supermicro systems almost exclusively. I know exactly why the story was written, and how the writers got it wrong. There was problem with Supermicro motherboards about 6 years ago. They used a IPMI chip that had bad firmware, which had a serious security vulnerability. IPMI is a feature found only on industrial-grade motherboards. it is a $1 chip that controls the main computer, basically a supervisory system running a stripped down version of LINUX with its own ethernet port. The system allows you to control the main computer, and when you connect to the iPMI computer, even if the main computer is off you can turn it on, update the BIOS, see the screen, and send virtual keystrokes and mouse clicks. So it is a spy system intended for remote maintenance. Dell calls it iDRAC, HP has it, IBM, and Supermicro has it. Dell charges $400 for the top level of IPMI, while supermicro is a more reasonable $100. Every motherboard from Supermicro with an F suffix has this feature. Anyway Supermicro fired one of their suppliers because their chip had very poorly written firmware and you could bypass the password, and take over the computer. We once had our IPMI port on the public ethernet, and it got taken over immediately and turned into a slave. Anyway IPMI is highly recommended to be behind a VPN, as you can completely control a system if you get into that.
As I recall, IBM was testing some software a few years ago, hired some “Contractors” to run the software through it’s paces. Fun started right away. Turns out, one new hire discovered a hidden folder, logging data, and sending it to a remote server. After reporting this anomaly, he was walked off the property and threatened with lawyers. So, I can guess this has bee common for years.
quit bashing China already.
the USA is even worse.
go look up five eyes if you don’t believe my words
We should all do better.
I have been a cautious Infowas listener for years ( before they even became known to the mainstream)
I have to admit, about 80% of thier predictions and “inside source” information about spying, and so called “conspiracy” has turned out to be completely true and admired/ proven true.
Typically, their lead time has been 6 month to a year. Espcialy telling was the NSA vault 7 release of spying details from Snowden, Clinton’s secrete email server and the now obvious “one party reporting” by MSM….all predicted way ahead of time by Infowars/Jones.
Today I found out Hackaday is a propaganda site.
Umm, this article is excessively biased. It:s nkt a condpiracu theory. Just another smear vsmpaign without need of hard evidence.
If huawei did the exact same thing as Cisco with s backdoor, they be banned. And you left out thst the alleged bavkdior in Huawei wasn’t a backdoor at all but TELNET and wrongfully drscribed by bloomberg.