This morning Bloomberg is reporting a bombshell for hardware security. Companies like Amazon and Apple have found a malicious chip on their server motherboards. These are not counterfeit chips. They are not part of the motherboard design. These were added by the factory at the time of manufacture. The chip was placed among other signal conditioning components and is incredibly hard to spot as the nature of these motherboards includes hundreds of minuscule components.
Though Amazon and Apple have denied it, according to Bloomberg, a private security contractor in Canada found the hidden chip on server motherboards. Elemental Technologies, acquired by Amazon in 2015 for its video and graphics processing hardware, subcontracted Supermicro (Super Micro Computer, Inc.) to manufacture their server motherboards in China. It is unknown how many of the company’s products have this type of malicious hardware in them, equipment from Elemental Technologies has been supplied to the likes of government contractors as well as major banks and even reportedly used in the CIA’s drone operations.
How the Hack Works
The attacks work with the small chip being implanted onto the motherboard disguised as signal couplers. It is unclear how the chip gains access to the peripherals such as memory (as reported by Bloomberg) but it is possible it has something to do with accessing the bus. The chip controls some data lines on the motherboard that likely provide an attack vector for the baseboard management controller (BMC).
Hackaday spoke with Joe FitzPatrick (a well known hardware security guru who was quoted in the Bloomberg article). He finds this reported attack as a very believable approach to compromising servers. His take on the BMC is that it’s usually an ARM processor running an ancient version of Linux that has control over the major parts of the server. Any known vulnerability in the BMC would be an attack surface for the custom chip.
Data centers house thousands of individual servers that see no physical interaction from humans once installed. The BMC lets administrators control the servers remotely to reboot malfunctioning equipment among other administrative tasks. If this malicious chip can take control of the BMC, then it can provide remote access to whomever installed the chip. Reported investigations have revealed the hack in action with brief check-in communications from these chips though it’s difficult to say if they had already served their purpose or were being saved for a future date.
What Now?
Adding hardware to a design is fundamentally different than software-based hacking: it leaves physical evidence behind. Bloomberg reports on US government efforts to investigate the supply chain attached to these parts. It is worth noting though that the article doesn’t include any named sources while pointing the finger at China’s People’s Liberation Army.
The solution is not a simple one if servers with this malicious chip were already out in the field. Even if you know a motherboard has the additional component, finding it is not easy. Bloomberg also has unconfirmed reports that the next-generation of this attack places the malicious component between layers of the circuit board. If true, an x-ray would be required to spot the additional part.
A true solution for high-security applications will require specialized means of making sure that the resulting product is not altered in any way. This hack takes things to a whole new level and calls into question how we validate hardware that runs our networks.
Update: We changed the penultimate paragraph to include the word if: “…simple one if servers with…” as it has not been independently verified that servers were actually out in the field and companies have denied Bloomberg’s reporting that they were.
[Note: Image is a generic photo and not the actual hardware]
so… hackaday on the bandwagon? whats up with this clickbait/fakenews thing? there is no proof whatsoever besides also the companies which are allegedly affected completely deny it and say bloomberg is full of crap
Exactly my thoughts. No proofs, but everybody fear behind BMC is helping to make endless theories.
Sounds like absolute example of FUD.
This isn’t a court of law and a blog is not a guilty sentence. Stop acting like the burden of proof is the same as if it was. The BMC SPI flash memory attack vector is VERY credible, and perfectly plausible.Chinese manufacturers are known to be willing to put in back doors. It has happened before; see almost any Chinese-made security camera. And the usual big American companies always deny and bury and play down their frequent security breaches, it’s the standard practice every time. You think the SEC or any other agency in the US would levy consequences on them greater than the hit they’d take if they verified this? At the very least their attitude should be to look into it, not release a lawyery outright denial using sneaky phrasing.
This type of thing definitely deserves investigation. In any case, the fact that this attack is so plausible already means that we don’t have properly secure hardware and infrastructure. If it didn’t happen this time, it’s only a matter of time until it does.
In some situations, uncertainty and doubt are perfectly rational and should cause fear. Stop using the term “FUD” as a thought-terminating cliche.
I want reddit stile up/down.
And gold. I give you gold today.
I too want those on HaD and hereby give you gold for suggesting it!
Its also entirely believable because we, the US, has done the same thing before. Its hilarious to think its clickbait when its pretty much a common, known intelligence strategy by multiple large states.
It one to know one…
It takes one to know one…
Indeed, browse through the crypto museum and you’ll see a rich history of devices like these. The DOD keeps a long blacklist of Chinese hardware suppliers in whose products they have found malicious hardware and software, such as a copy machine that sent a bitmap of every document scanned to a server controlled by the Chinese military. Ideally large publications should do these kinds of exposés for all supply side attacks in order to educate laymen on the importance of secure hardware, but they have to start somewhere.
Disguising a malicious component as an 0201 passive on the data line is very possible. It could just count the bits moving from the firmware flash memory to the lights-out SOC and inject its own bits at the proper moment. Rootkits for this type of system can be a few kilobytes, perfectly feasible with our current memory density. Obviously probably not a component that you can just pick up on aliexpress, but few military intelligence bugs are.
If the subject wasn’t so scary, your posting would be pure pleasure to read. Bravo!!
I want to find out more. The one photo of the alleged component was a 3 lead device which could possibly hit an SPI line depending on the mode a BMC would be running. I don’t know about Supermicro but more than one company I’ve worked for signs their BMC firmware and check the signatures before loading. Using a hardware/firmware separation system along with cyptographically signed images is one way companies have been able to stave off potential issues of Chinese implants.
Some basic industrial inspection should be able to clear this up. The companies have board layouts they can use as a baseline. Deep learning might accelerate the process but there are plenty of imaged based inspection systems using more traditional computer vision techniques.
I’ve been pointing out the BMC every time the editors here would scream about the ME. Looks like someone finally got their attention :P
3pin device? where does the gnd and vcc goes?
VictorT: Many low-power ARM processors will run off the power on the data lines alone if you let them. In fact, when they debuted the very first ARM, they accidentally ran it on a faulty board and were delighted when it still booted anyway on power from i/o. Of course you or I would never intentionally design a circuit that way, but we aren’t spies in the People’s Liberation Army either. It could certainly help them reduce pin count and avoid having to change the board layout as much as possible.
Not that the actual bug is a whole ARM processor, it’s certainly far more simple than that. My point is that this is not a traditional design with a traditional application–it is a very niche design with a specialized application. It’s not right to assume it will follow the same design rules as the circuits we build. I’m willing to bet they managed to make a device that runs entirely off the tiny bit of power available on the data lines, is disguised as a normal passive you’d find there, and probably fulfills a legitimate role when it isn’t pwning your machine.
There was an article right here on Hackaday a while back about the history of ARM and how their first test bench neglected the VCC and ground pins.
https://hackaday.com/2018/05/08/sophie-wilson-arm-and-how-making-things-simpler-made-them-faster-more-efficient/
This image from Bloomburg may be garbage but it shows a 3 lead device.
https://assets.bwbx.io/images/users/iqjWHBFdfxIU/i9VdsjZLS_Pk/v1/-1x-1.jpg
I’m not sure what BMC SuperMicro was using, perhaps the Pilot 3 or 4 but the datasheets are locked behind NDAs so it’s hard to say what bus they might be hitting. I’m starting to think this story will fall apart with more prodding. “Extraordinary claims require extraordinary evidence”, and I’m not seeing too much evidence here. I’d like to see one of the alleged devices in Chipwork’s hands for some die imaging (assuming the device is indeed a microcontroller).
Isn’t that only one side of the chip, ie we can’t be sure it’s only 3 pins. Though even a three pin device with scavenging power can contain a low spec minimal processor runny off a shift register with even as little as a few megabytes of eeprom. Go to even smaller geometries during fabrication you can increase to do something like 100M.
Not at all difficult for a committed well funded team from design through to fabrication. Also bear in mind the device could well be positioned vety closely over a data line and designed such that its base thin as possible with appropriate dielectric and pick up something useful from electrostatic effects ie with fet opamp etc thus leaving the visible 3 pins for something else – even making them look especially benign just to keep the inexperienced off the track and so easily too :D
+1
Well put, mirrors many of my own approaches to risk assessment such as the USB memory chips which appear as say 4G but have another 4G hidden that collect personal data IP etc exploiting the USB processor ostensibly flash reliability etc At some point such as with virus scanners then upload the data suitably encrypted to appear as system reports…
Wow, a Chinese memory stick that has MORE capacity than it claims to.
I think you missed the point or satire, you can’t (normally) access that extra memory :/
Once you understand the economics and reliability issues of having a processor there on shuffling data to maximise data retention longevity overall then you see it works out more cost effective to have say 8G raw bytes with anything from 1G to 4G or so as the specified amount for consumer access. The remainder selectively used to manage degradation in ideal cases with the data shuffling not evident only the occasional delay a bit longer than normal can give it away (which I think is how it was stumbled on initially the cluely investigated such delays) or store ‘other things’ for the less ethical though easily accessed.
Jokes and satire aside it has been observed by various groups and focused experts in USB flash who have the capabilities and equipment to definitively investigate – it Has happened and easier than first discovered for moderately skilled hackers to extend upon and at more sophisticated levels !
For some populist though pertinent posts do a search on Phys.org or physorg.com from about 5 years ago or so, there are also other forums though mostly paywalled which go into rather more detail, not difficult too for people to reflash the embedded processor or add an extra comms channel out the other side of the memstick for all sorts of uses :-)
You can extend the paradigm to all sorts of other peripherals too…
Yes I know what flash over-provisioning is. I was making my own little satirical reference to the many cheap USB sticks found on places like Ebay, where Chinese sellers have tinkered with the drive’s identity info, causing it to report as, say, 16GB, even though it only contains, 4GB of actual flash. Works fine for the first 4GB, but once you try writing more data, random bits of the old data are wiped out.
As for your hypothesis, sticking extra hidden memory into a USB stick wouldn’t really be a threat to anything. To collect whichever personal data, you’d need access to it, which is a huge and different kind of problem. Where you put the data once you’ve stolen it doesn’t really matter, a PC would have plenty of room on the HDD. It’s getting the data in the first place that’s the trick.
Okie Greenaum :-)
In respect of your last para its so much easier than you suggest and there are many ways.
With an appropriately configured cpu in the usb memory stick it can be accessed with only a few keystrokes native on the system in a dos type shell when you know what to look for. Worse it can also be easily accessed by so called virus scanner software from third party sources abd either as original software well marketed or as commercial software patched to seek particular responses easily obfuscated as don’t care aspects of windows semi-automatic rescan of connected storage devices and OS logs too though not essential.
You have to consider that the plan to just put in hacked memory sticks is only one dimension of the tactic of an overall strategy. IOW. As part of a plan and knowing some basic OS exploits (most) for storage type devices in windows is dead easy as the ‘signals’ and cues are already set up in the memstick’s cpu’s program to enable it in the first place when first recognised by your system upon first program and ‘format’ or error check ;-) Hmmm…
Indeed and this seems to be a good writeup re the BMC
https://www.electronicdesign.com/embedded-revolution/how-hack-server-motherboard
It doesn’t read like the kinda thing some reporter would be able to simply make up, also consider that it’d been previously reported that Apple found malicious modifications to SuperMicro firmware that they also denied https://arstechnica.com/information-technology/2017/02/apple-axed-supermicro-servers-from-datacenters-because-of-bad-firmware-update/
It’s much more likely that they’re all under a gag order
Negative.
This sounds much more like something that escalated out of hand. Someone found an extra component that was not expected. It could say be an extra signal protection device that was added “last minute” but something with SCCS went wrong so that the “current” design no longer shows it. And from there things escalate: You found something unexpected? Could it be that this is a spy-device? and the engineer answers: “It seems unlikely, but it could be”. And then another engineer gets asked: “What COULD you do with a tiny chip implanted into a motherboard?” Again a truthful answer and with the right twist this is blown up to what we see now.
Exactly. Something similar to that guy that started saying the passives in a ethernet magjack connector were there to spy on him.
One wire comms protocols, the holding up or down of a single bit that controls some memory access rights (at the desired moment). There is also scope for a three pin device to be an entire cpu core. Just knock 3 pins off a PIC10F for example.
Well yes, if you put it that way, journalists ARE lying scum. And have a completely different idea of truth, knowledge, and implication, from the scientific or engineering versions. It quite possibly *could* be sending your holiday pics to Kim Jong Un, but as all smart people know, without a likelihood attached to that possibility, it’s a worthless thing to say.
Yeah, they deny it. And?? You’re going to take them at their word? When it’s likely that they would deny it to try to protect their business from the negative publicity from the knowledge that they were hacked? Are you really that naive? https://www.washingtonpost.com/news/innovations/wp/2017/11/22/uber-paid-off-their-hackers-and-theyre-far-from-the-only-ones/?utm_term=.3ea5f9a75115
Yeah? Well, I deny that they deny it! Now what?
quo bono?
Bloomberg? The liability of posting a flat out fabrication with implications capable of harming stock prices could trigger a future SEC investigation. Probably not in their interest to do this.
Apple? Has a reasonable stake in denying this now and producing a full accounting later to forestall judgment, let the issue cool down, and then show they’ve done their homework to limit the scope and impact.
Amazon? Raise your hand if you haven’t used an Amazon product, hosted service or service. They have plenty of reason to punt this issue with short term plausible deniability.
China? Countless articles, sources, and history shows their desire, capability, willingness and tools to do something like this.
Sounds pretty credible.
Now all we need is the Butler with the murder weapon.
It is since China is very big into IP theft and hacking our systems both military and civilian and has been for decades.
Kind of makes it seem like it was a bad idea to give our whole economy to them, when you put it like that.
This sounds pretty credible as well. But it’s very much NOT what you think you’re hearing.
https://www.eleccircuit.com/cricket-voice-with-buzzer-using-lm324/
Any company that knows how to handle PR well in any case would never disclose something like this
Re: “on the bandwagon”. Nope.
We thought this was an interesting and hacker-relevant article making amazing claims, and at least claiming to have backed them up with many (anonymous) sources. We also note that Amazon and Apple denied the meat of the Bloomberg piece.
We don’t know any more than you do. Details will come out or this will fizzle away. Who knows?
How did you get the impression that we were passing judgement on the meat of the article?
Probably way how this story will unfold might not be pleasant for all those who think this event took place, as it caused significant drop in stocks, Supermicro at least.
There is no obligation for obvious FUD journalism that is based on “anonymous trusted sources”, but only while you are in “civilian space”. When you get in spot of SEC – things change.
There is a lot of examples when journalists acted irresponsibly, but hackaday usually is very responsible in this matter. Can’t say about this article, article is not neutral and slip on side “it is likely possible”, and not much credible technical details why it is possible or not, and how, which i expected from hackaday, and i believe this is why many people disappointed.
Sorry if you thought that we had vetted (or were even able to vet) the reporting on the Bloomberg piece. I wish! Someone send us a board?
That stuff about the SEC is pure speculation, though, right? Or do you have information about illegal stock market activities?
About SEC,expectation and, yes, speculation based on this 5 years old article: https://www.businessinsider.com/bloomberg-reporters-compensation-2013-12/
Usually when repeating a story like this, credible people add a word like “reported” or “allegedly” in the title or headline.
Oh man, I would love for Hackaday to get our/their hands on one of these boards. The folks over at /r/homelab are already on the case: https://www.reddit.com/r/homelab/comments/9lapzs/big_supermicro_hack_how_many_of_us_bought_these/
If this is true, someone will be able to verify it in pretty short order. It’ll be a feather in the cap of whoever’s able to confirm it first, so you better believe a lot of eyes are looking.
pssst, hackaday started sucking when they were bought out by large media some years ago, incase you missed it
They were bought by Supplyframe, who sell electronics, I think, something in the hardware game anyway. And before that they were founded by a well-known media douchebag. They do OK really, considering.
“But the company SAID that they weren’t compromised, so they must not be!”
Just like Intel said their speculative branching was absolutely secure, right?
When in doubt, dike it out.
This article is pretty much just a fluff job until we know which motherboards are affected and (hopefully) how we can remedy this.
According to the article, major commercial purchasers of the affected boards were notified in a secret meeting by US officials.
So minor commercial purchasers are left out in the cold? Even though they pay tax money as well?
This sounds a bit like how Meltdown was handled. I hope this doesn’t become the norm for security discoveries, particularly hardware ones where foreign states are involved.
“It’s ok as long as we notify the top 60% of users, right guys?”
also upon further reading this particular article of HaD i see that this is a talk of theoretically talk, but i mean i can also say theoretically i will be able to beam and use a replicator in the future if i am alive and if it will be invented/made reality
“Bloomberg also has unconfirmed reports that the next-generation of this attack places the malicious component between layers of the circuit board. If true, an x-ray would be required to spot the additional part”
hell why bother with that? just implement it in the Silicon of any cpu i mean its not like its not possible, they could go further and make their fantasy talk more ridiculous and say its in the silicon wafer at manufacturing process. so that will be like impossible to check then
Considering the parts aren’t all sourced from the same manufacturer, sneaking malicious silicon into a cpu is probably not going to happen unless the chips being sourced came from a pop up Chinese fab house turning out fake silicon. It sounds like someone at the board factory, or someone in the chain or custody, is going to jail. Nobody is accidentally installing a chip that isn’t in the BOM, unless it’s a clone part disguised to act like a specific component (like an emulated chip with “enhanced” functionality). The board design was either intentionally altered before it went to the board facility, or it was done by someone at the board facility.
Like an undocumented “extra feature” in a clone of any of the i2c temperature sensors that happen to live on the same multi-master bus segment as the BMC. Yet one more reason management processors of all sorts are a terrible idea.
“…a pop up Chinese fab house turning out fake silicon…”
They have the resources, the knowledge and the ability. Maybe not a current-generation Intel part, but how about an ARM that looks identical to the genuine one, but includes a backdoor for PLA hackers?
It will be interesting to see where this leads. The Bloomberg article is persuasive, but a marked up schematic of the motherboard in question, showing exactly where the mystery chip was connected would be the nail in the coffin. It’s hard to be 100% convinced when all the principal players are saying “no comment” and remaining anonymous.
this is yet another reason why open hardware is essential, and why we need a worldwide repeal of IP law.
This wont defend you, if for example they will embed custom uC inside your flash chip, that will modify your boot scripts or add new user.
Huh? How do these relate? *IF* this attack actually exists, it is just a chip pretending to be a capacitor or a resistor on the board. Explain how open hardware or IP laws would prevent this.
I guess open hardware and repeal of IP laws that prevents you from doing so means you can choose your manufacturer yourself, and choose your hardware components yourself, which means it’s possible to be more careful about what you put in your computers. Market pressures from people avoiding manufacturers found to have behaved poorly can eliminate some significant percentage of manufacturers doing this, and diligent people can avoid the problem with their own hardware.
correction: s/prevents/prevent/
Lol. sed won’t save this!
*looks at forum title* IP doesn’t prevent THIS crowd from developing their own hardware.* Economics do.
*Lots here do it for fun, and employment.
Until your operation gets big enough to get noticed and then the big guys who SAY they have IP rights hit you hard with a big “lawyer stick” and destroy your “sand castle”. IP laws do get in the way of people choosing their sources. If everyone was permitted to make anything we could restrict our purchase sources to those we trust or want to favor without having to chose the inferior product or, worse yet, going without. We could have our favorite producer make each product or if we are really tin-foil-hat paranoid, (sometimes appropriate) we could even make it ourselves.
No. This wasn’t a problem when our computer systems were designed and manufactured here in the U.S. by Americans,.
But we cut a deal with the devil, we wanted cheap electronics and didn’t care what it cost in human, environmental or security terms.
“No. This wasn’t a problem when our computer systems were designed and manufactured here in the U.S. by Americans,.”
Components weren’t quite so small and easy to hide back then.
Also there weren’t so many that it would take a huge team to trace out a schematic of the board. Schematics were also easier to come by with libraries all over the place keeping them on shelf. Tracing out schematics was also a more widely known skill with how useful it was to fix radios, VCRs and TVs. Skills that seem to be slowly making a comeback with recent increase in interest of hacking and security.
You think the NSA doesn’t have similar capabilities?
In fact it’s known that they do, which is why it won’t surprise me if this story turns out to be true.
https://arstechnica.com/tech-policy/2014/05/photos-of-an-nsa-upgrade-factory-show-cisco-router-getting-implant/
At least that’s in transit. When your manufacturer is working against you it’s a whole other ballgame.
The logistics of detection are hard. Even if you got the blank PCBs and electrically traced/tested them all to check continuity, you can’t then ship them back out again to have parts put on them. You can’t know that those boards will be the ones used.
The best I can think of to mitigate this is you have to do the final assembly in house or somewhere similarly trusted. That means that anything you bring in has to be testable or perhaps too small to practically subvert without detection. (passives, unpopulated PCBs, maybe simpler logic devices, maaaaybe things like SPI flash chips) This works against integration though, unless you can get/make the tightly integrated parts yourself. You then can’t send it out again somewhere untrusted for final or late finishing.
This was mentioned actually in the article on the electronics tariffs, since without them companies would prefer to do final assembly at home to avoid leaking IP. (a similar concern)
I’d be interested to hear if anyone else can think of other mitigation strategies. here are a few half-ideas:
-One thing about unpopulated PCBs is that all the major components that *should* be there have pads on the board but aren’t populated yet, so you should be able to electrically detect anything inside the board that tries to tie to them. They can’t *not* put the pads down for these major components either, so can’t deny you a solid place to test from.
-For intermediate complexity things like flash chips, you may be able to use use volume manufacturing against your attacker-manufacturer. A flash chip may be subverted to modify a bootloader, but that probably has to be customized to the bootloader. you can’t put human creativity in the device. That means if you can scramble who gets what components (say, if different downstream companies want to mix and match stock with each other) then subverted chips may be more difficult to send to a particular buyer, and detection may be more likely.
-For very large scale integration components (like CPUs) they’re harder to modify in transit. IF you can trust that the CPUs you got from Intel are good (Intel ME, haha) because they make them themselves, then you can be pretty sure they’re still good after going through a few manufacturing steps elsewhere. Of course, that may means you can verify the CPU after it’s on the board, but you can’t verify the board because it has a CPU on it and you can’t get to it’s pads.
I think the PCB point in particular is interesting and could be built on. Major components in a design have to be there and can’t be hidden without detection. In a way, it’s like the manufacturer can’t carry out a chosen plaintext attack. That also means that whatever debug interfaces you put in have to be there to avoid detection (failure to debug raises a security alert). Then you just have to make the correct debug output practically difficult to emulate to avoid them routing the debug somewhere else.
The more I think about this though, the more I think that you’ll always be fighting a loosing battle.
Erm… you would have just had a chip by Dallas Corp. Seriously.
My first thoughts were of a 3 pin Dallas Onewire Silicon Serial Number flip chip I used around 1999.
“But we cut a deal with the devil, we wanted cheap electronics and didn’t care what it cost in human, environmental or security terms.”
Yep. Exactly the thought that the writer of this article wanted to put in your mind. :)
Mark my words. The next article from Bloomberg will be called “Why you should buy American”.
This hack and others with hardware implants work by exploiting outdated firmwares in the BMC, if the BMC was open source firmware or had the ability to run open source firmware then it could be installed in the USA by the company making the servers after supermicro makes them rendering any hardware implant designed to exploit the stock firmware harmless.
Once you modify the hardware, anything you do in software goes out the window. Hardware lies, and software is composed of the lies.
An implant or other manufacturer-done alteration can subvert ANY software mechanism. They could backdate the BMC. Or patch it’s firmware with a new exploit. Or just man-in-the-middle it. Or disconnect it and take over it’s functions. Or any number of an infinite range of other possibilities that may not touch it at all but modify/subvert other components of the system.
Encryption and signing can solve many of that sort of problem. Or at least limit the possible culprits.
You could disconnect the BMC and take over it’s functions but that’d require a lot more than 3 pins. And, again, if the other motherboard chips demand signatures and use encryption, you can remove that problem too.
Not that signing is an innately good thing, it just means that if, eg, Intel, want to fuck you, not only can they, but you’re powerless to stop them.
I dunno, I was happy with 8-bit computers. Nobody ever hacked them. Bring them back. They even do SD card interfaces for most 8-bitters now, with those we could dig out the old modems and reimplement USENET. It was the only bit of the Internet worth a fart anyway.
The remaining Internet can be streamlined into an efficient pornography distribution system and that’s about that.
Given that none of us can build our own PCs all the way from silicon to kernel, open source is pretty useless, as we’ll have to rely on someone else to supply us with parts, somour trust is gone. And that’s assuming you can trust the open source design, and it’s not been carefully compromised.
But open sourcing the CPU designs would make it trivial for every state actor to produce their own clones with added features, so at least it’d level the playing field. Perhaps the PLC hack on my BMC would fight against the Russian one in my CPU and I’d end up safe?
“signal couplers”? WTF is that supposed to be? When somebody fits an ARM server in a pair of 0201 0.1µF AC coupling caps, let me know. I want to shake their hand.
And buy a couple to use in some projects.
It is pretty small, the Bloomberg article has a photo.
https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iNO3klzCOEjQ/v1/-1x-1.jpg
LoL at the irony… “In God we *trust*”…
Have “liberty” just below “in god we trust” is already irony :D
To be fair, Lincoln is glaring at it pretty sternly LOL
I prefer the film “In God We Tru$t”
I’d sooner trust God than most humans. (Calvin was right. – People are born as nasty, selfish, and even evil souls. They need to be taught to be good.) With God, we have a chance. With carefully selected exceptions, I wouldn’t want to trust anyone else.
I don’t think that’s the actual part. I think this is just a photo because articles need photos. This is why we added a note at the bottom of the Hackaday article that this is not the actual hardware.
Articles MAY be more attractive with photos, but HECK – we read entire books with no want of them. BUT to NOT make an under caption or first line akin to, ~”representative and not actual photo,” is a huge “fart in our general direction.”
Fair enough, I thought the disclaimer was for the HAD photo and not necessarily the Bloomberg article but it makes sense that this isn’t the part and it’s just a pretty picture for the article. I got excited and my naivete got the better of me.
That looks like an 0201 RF balun. Plus if that’s an actual extracted part, they went to a lot of trouble to clean it up and make it look like it came right out of a fresh tape…
I use a TDK balun that looks exactly like that (6-pins, white ceramic, black square indicator). Part# HHM1591D1 (0805 size)
Yup: https://www.mouser.com/ProductDetail/TDK/HHM17165A1?qs=sGAEpiMZZMtMMXztyU6kdA1NDmnec753KB5dmlMIDZBDd6Te8ypetA%3d%3d
What business does an RF balun have on a server that doesn’t have wireless?
I’m sticking to my original assertion that there’s a stock scam going on behind the scenes.
Contemporary motherboards operate at RF, so Baluns do have a use re EMI/RFI and EMC susceptibility issues. It’s the placement which would betray a lot such as corrections for impedance discontinuities eg thin power lines to more susceptible regions, dielectric issues again re reactive impedances, cleaning up jitter and noise etc Versus closest access to data lines.
Further to my earlier post, it wouldn’t be hard to put unethical circuitry in such a device of that size then make it look exactly like a ccommercial Balun and place in an appropriate location, so depending where it’s located with some analysis of just what its close enough too as the earliest start of analysis can tell heaps from simplest forensics :-)
I have no idea if the pictured chip is meant to be the actual malicious component or just a visual demonstration of how stealthy these components can really be, but bear in mind it would make sense for it to look like a passive you might find on a high-speed data line or whatnot. Obviously if they were disguising it as a passive, they would make it look just like a passive that would normally be there between the firmware memory and the SOC it is attacking.
Looks like a diplexer i used in a previous ISM band radio design.
https://www.johansontechnology.com/datasheets/2450DP14C5400/2450DP14C5400.pdf
If they can’t report with the correct term for the actual part for this hack, you can pretty say that they don’t have someone in the technical field for doing fact checking. So far, I have not seen a second reliable source collaborating this “discovery” nor showing the actual component that is soldered on the motherboard.
I would prefer to see a more technical review of this in a security oriented website than that of a financial news outlet. HaD is just throwing more oil onto the FUD.
Bloomberg is not a tech publication. They are using a term for non-tech readers. We are not really the target audience.
FYI: This is a more technical look at the alleged claim by a 3rd party that knows a lot more about servers than a financial publication. Wish HaD use this instead of the bloomberg article.
https://www.servethehome.com/bloomberg-reports-china-infiltrated-the-supermicro-supply-chain-we-investigate/
I think bloomberg needs to be investigated for SEC violations (market manipulation) as several publicly traded companies lost market share because of this report. I wonder if someone at Bloomberg shorted these companies before the report went public ….
I don’t really get what this article is saying. Bloomberg said the chip was sitting on the addr/data lines of the BMC. Why is everyone acting like Sherlock Holmes and saying “whoa, Bloomberg was LYING: it’s actually a chip affecting the BMC that they are talking about!”
Might well just be an op to raise defiance on Chinese made hardware and “buy american”
> HaD is just throwing more oil onto the FUD.
Agreed.
Um… So can you like, yell at someone?
The use of the term “FUD” has become something of a red flag for me. Some things in the real world deserve to be treated with fear, uncertainty and doubt. The cultish use of this quip to instantly ridicule and dismiss those who have any reservations about something–especially crypto, the coinheads love this term–is really unhealthy and almost a form of newspeak.
First of all: no FUD. We reported that Bloomberg wrote the linked article. We don’t know any better than you whether it’s true or not. We want to see real tech details before making any decisions.
Some of us _do_ think that attacks like this are plausible. Look at the NSA’s leaked ANT catalog, and then recognize that it’s ten years old. We talked to two security Joes (Fitz and Grand) and they think it’s right on the edge of possible. This doesn’t mean it happened, but it doesn’t mean that it didn’t either.
Anyway, it’s fairly clear that there’s a _lot_ of information missing here that we’d all like to have. Above all, if this hack was carried out, we’d love to see a good writeup! If it ends up being a hoax, maybe there’s a story there too.
> We don’t know any better than you whether it’s true or not.
Then don’t write a headline that suggests otherwise.
Software side of the story.
https://www.bloomberg.com/news/articles/2018-10-04/the-big-hack-the-software-side-of-china-s-supply-chain-attack
Journalists perform lossy compression on information. We don’t know if the lack of detail is because they simply don’t have it, or if it was edited out because most readers wouldn’t understand it or have any use for that much detail.
“ournalists perform lossy compression on information”
LOL!
(so true!)
“The map is not the terrain” but we do what we can.
Oh, I wasn’t referring to you [Eliot].
I once had the local newspaper do a personal interest article about me.
When I saw it, I was surprised at how many mistakes were made about what I had said during the interview.
(I wish he had let me proofread the article before sending it to his editor.)
“Journalists perform a logical inversion operation (Boolean NOT) on facts” is another nice technical metaphor.
The words “malicious” and “found” have pretty clear and specific meanings.
The ARM server isn’t in the attack chip, the ARM server is on the lights out management system which allows you to remote into the system in question and restart, or change bios settings from across the world.
Are you deliberately misrepresenting what’s being said here? It’s not an entire ARM server. It just intercepts and modifies a few bits of data coming into the ARM BMC from flash memory via SPI. Depending on how economical you are with your code, you could definitely fit enough into that package to trigger an exploit in the BMC.It doesn’t need to run linux on its own or anything.
Perhaps Bunnie isn’t so crazy for making a custom laptop from scratch.
Do I remember him saying somewhere that he was seeing strange network activity from the motherboard management systems and he was thinking that it was backdoors designed into the management system… Maybe he was right for wrong reasons, was seeing things not designed in by intent of the originator but tagged on during manufacture.
Those who are paranoid about infosec are always proven right over time. It’s not like most other types of conspiracy theories which are always tinfoil and hot air. This stuff really happens, and happens often. It would be foolish of us to think the country we offshored all our IT manufacturing to wouldn’t ever try something. And this isn’t the first time.
> Those who are paranoid about infosec are always proven right over time.
Pre-Echelon, we thought phones were reasonably secure, too. The resources necessary to monitor everything simply wouldn’t be practical.
And then CALEA happened and every two-bit skiddie got into the switches and listened to whoever they thought might be funny, because a backdoor for one is a backdoor for all.
Pre-Snowden, even the craziest most paranoid whackjobs placed limits on their speculation. “theoretically, the NSA could infiltrate the SIM card makers and log all the Ki values being programmed in, and not even need to break encryption, but realistically nobody would do that, it’d carry too much risk of discovery”. And then it turned out that even the most paranoid whackjobs were only able to anticipate a small portion of what was actually true.
And now we’re seeing that our own ANT catalog isn’t the only offering…
“China’s People’s Liberation Arm.”
China’s People’s Liberation ARM
FTFY
B^)
Also LOL at all the Apple fans surprised Apple is running SuperMicro baords instead of custom internally designed Xserves
Are you kidding me?
The last XServe used Xeon 5500 and Sata 1, you would have a hell of a time trying to run a cloud on that…
They actually made some nice-looking server hardware. If they still made common-sense normal server gear I would probably own some. They’d have that nice aluminum sun/oracle look without the kidney-sized price tag.
“Bloomberg also has unconfirmed reports that the next-generation of this attack places the malicious component between layers of the circuit board. If true, an x-ray would be required to spot the additional part”
hell why bother with that? just implement it in the Silicon of any cpu i mean its not like its not possible, they could go further and make their fantasy talk more ridiculous and say its in the silicon wafer at manufacturing process. so that will be like impossible to check then
this was actually written about already. Even already manufactured devices can be “modified”.
https://semiengineering.com/every-chip-can-be-hacked-with-this-tool/
There was also a story about a photomask design being manipulated to create a control signal backdoor. I cannot find that article though.
You mention that it’s perfectly possible, yet in the same sentence you refer to it as “fantasy talk.” Is this how your logic works? The People’s Republic of China has proven time and again that they’re willing to sneak back doors into the hardware they manufacture for us, the attacks are very plausible from a technical standpoint, and our security firms rarely seem to check or care. You think that’s not newsworthy? You think it’s so outlandish that it shouldn’t be reported on?
You know that PCB embedded components are a real thing, right? It would be trivial to hide a malicious component disguised as an embedded passive between a memory chip and a processor.
If this is true it is very scary. We need Open Source Hardware.
Contract manufacturing/outsourcing is the boogeyman, NOT closed source. A hack like this can still happen in completely open source system if your board house planted such an alleged chip. The open source nature makes it much easier to understand your system and replicate the board.
>The open source nature makes it much easier to understand your system and replicate the board.
and check the finished product because you have full BOM and schematics. That’s my point.
Against the resource of a nation-state, you’d need comparable resources.
Are you going to individually desolder all the components one by one, match them to the BOM, Xray them to make sure that there are no hidden part, xray the multilayer PCB to verify that it agree with the gerber files next time you buy a computer? The alleged chip looks like a passive component soldered into the board that you might not see.
If the board is manufactured in a trusted contract manufacturer with proper auditing, trusted supply chain with bonded carrier, incoming component inspection, then this is not likely to happen – Open source or not. I worked previously in defense contractor and that’s exactly what we have done and we have our own in house manufacturing facilities.
It’s disguised as a passive that is supposed to be there, and probably also does that passive’s job in addition to delivering its payload. Checking the BOM wouldn’t find it. When was the last time you checked every little SMD part on a board with enough rigor to find this?
In the grand scheme, we shouldn’t even be worrying about whether this specific report is true or not. The simple fact is that it’s possible, so we can be utterly certain that somebody out there is going to do it.
If I’m contract manufacturing, I have the designs and BOM, because it’s my part I’m getting them to manufacture.
Open source only makes it easier for the bad guy. Closed source is only security by obscurity here, but that’s about all we’ve got now, if you can trust even passive components.
What good would it do ? As per the article, the problem is the factory in China modifying the board and inserting that teoretical component. If it wasn´t detected in the posterior inspection by Supermicro, what good would come from it being open source ? The person/entity that ordered the boards would probably not detected that teoretical tampering either.
As for the article : a better tone for HaD would be to report that there were talks about this, but not present it as a consumated fact without the backing evidence.
Next in this channel : flying saucer from Jupiter lands in the North Pole !
“What good would [open hardware] do?”
One of the biggest problems I have with purchasing hardware is getting the specific set of features I want. Once a particular general form factor becomes widely used, manufacturers join a race to the bottom, and pretty soon everything looks very similar in terms of feature selection. That means that when you want something a little more niche for feature set, you often get funnelled into only having one make-and-model of device, period, and you get that XOR get to make some choice based on the supply chain leading to the end product, not both. At least with open hardware it should become easier to get both the features you want and some control over the supply chain. With a little work, you can gain some slightly improved assurance nobody in the supply chain is pulling a fast one like what this article describes.
That’s what good it would do.
Wrong. Not about the importance of open hardware to one´s specific case of features. If you contract manufacture to others, thei can do whatever they want to your product and, if they are competent enough, you will not discover it. As someone said in other places, wiill one xray all of the boards and components after they are assembled ?
As for the features, I understand you . PS/2, COM and Parallel ports should not have disappeard from most motherboards. Ditto for IDE connectors. There are many things that I have never seen used that keep coming in new motharboards, while some useful common think like a PS/2 keyboard now needs a converter to USB.
>Next in this channel : flying saucer from Jupiter lands in the North Pole !
Not Jupiter invasion. Just tourist. Accusations of spy asteroid in polar orbit most obviously fabricated. Went to see worlds famous Mr Claus and have photo opportunity with his many helpful elves. Looking very forward to seeing stomach wobbling like bowlful of jelly and light up nose of also very famous Mr Rudolf.
Unfortunately snow was unexpectedly slippy but hope to come back soon.
The point is to make it more difficult to pull off these attacks without detection. It wouldn’t be impossible, but it would probably reduce the frequency of attempts, and I bet they’d leave a bigger paper trail as well on their end.
Seriously, I know spy technology and all that crazy military R&D, but what can they have possibly packaged into a component the size of a 0603 capacitor that would do any kind subvert operation? I smell bullshit and I am waiting for El Donald to start tweeting about it.
Yes, and further, how does the manufacturer have the knowledge or ability to add traces and a component to an already crowded PCB, which will not affect normal operation of the device (and pass testing) but also somehow allow it to “ping anonymous computers on the internet”?
The only thing I could fathom is a chip that is a Trojan horse. So basically, it appears on the outside to be a signal coupler(?) and it’s pin arrangement is compatible with the existing pad footprint, but it emulates the original chip while being able to inject or fabricate signals in-line. It probably has a built in oscillator and it possibly leaches power somehow. I won’t dismiss this as hogwash just yet, there could be some truth to this. I mean, if this component is in the signal path, it has a way to inject data. This reminds me of a modchip for a console, except its disguised.
Read further down. Later versions were inserted between board layers. How is that done?
Note: *unconfirmed* report…
You can make a cavity in the inner layers in the PCB, put the part in before the whole stack is laminated under heat and pressure. google “embedded PCB components” and read the first few results.
The chip and antenna inside your credit card is done in a similar manner.
Hiding something between pcb layers is the most stupid idea I ever read about.
With the first hardware security audit there will be one wondering why the hell the developers choose such a difficult manufacturing step to build only one tiny piece of hardware into the most difficult area one could choose.
Might want to recalibrate your brain then. Embedded components in PCBs have been around for a while.
You only have to compromise the PCB fab house this way, rather than having to compromise the fab house with updated gerbers AND the assembly house with new parts, which now involves a whole other companies supply chain, and the corresponding paper trail.
You want to read precisely: “build only one tiny piece of hardware into the most difficult area one could choose”
About the only way I can see this working is if it was a really tiny microcontroller hooked on to one of the UART pins of the motherboard’s platform controller. Then it can console in and inject a script into the Linux-based ARM system in the platform controller. I guess it wouldn’t be too hard to do if you couldn’t change the root password of the platform controller?
All it does is sit on data lines of the remote mgmt chip (think Intel ME, or similar on-board ARM micro?) and twiddle with the inputs. It’s a dumb device but supposed to weaken the target ARM, which can then be triggered to do malicious things. I guess you can make a Game Genie pretty small these days.
That’s along the lines I was thinking – maybe they discovered if you short two traces to the IME-type gadget with a resistor, the security is compromised and remote access it enabled.
Security researchers have been preaching the dangers of remote management chips for years. They’re useful especially in a multi-server setup, but I don’t think most are convinced due diligence has been done in their creation.
Good metaphor, actually. Very good!
Take a look at the playstation 1 modchip:
“The data that the CPU is looking for is a serial data stream at 250bps
consisting of the characters SCEI, SCEE or SCEA depending on whether the
console is Asian, PAL or North American. By sending all three data streams
in a rotating sequence, the chip can satisfy the console that it is reading
a CD of the appropriate region.”
It does this with a small microcontroller sending out the sequence on a serial interface.
A comparator, clock, and a few kilobytes of memory. Maybe a buffer. All it needs to do is count some bits coming from the firmware chip and insert its own little binary blob at the proper time. We have 256 Gb on micro SD cards–this is actually not only plausible using current tech, it’s almost trivial. We’ve seen comparable components that can actually run off the power in the data lines alone. And this isn’t even getting into the more exotic stuff that a state-level intelligence agency should be capable of. It’s not like this little SMD needs to run linux or anything.
I see a lot of people saying they “smell bullshit” but really it’s more of a lack of imagination or effort to understand. Most of us are not accustomed to designing for these purposes and the restrictions that go along with them, but those hardware hackers who do will see this is perfectly possible and in fact the information available in the Bloomberg piece actually lends credence to the idea that it was received from a real expert source, and not invented yellow journalism or exaggerations.
When we design something we do it the “right” way because it’s better, it’s easier, it’s more robust–not the sneaky way that you have to do things to get way with a supply-side attack. I’d just like to point out that’s just enough knowledge to be dangerous; you think you know the subject, but it’s actually quite different than what you know. And FWIW Bloomberg is not particularly in support of Donald or his trade wars, so I doubt it’s about that.
That would mean the PCB layout, PCB manufacturing, and Assembly house conspired to implement this hack …
Well if said PCB mfgr, and Assembly house are in the same country owned by a secretive government…
“Just because you’re paranoid, doesn’t mean someone isn’t out to get you!”
B^)
i guess there may be a limit to the length of the post so here is a link to the text
https://pastebin.com/n2br4RQY
Also, are we meant to believe that they fabbed a custom cloned PCB for those server hardware just so that they can add the extra connection for that super secret microscopic component? Because I can’t see how they would be able to just drop in that component in the spot of a smd capacitor or resistor and have it do anything useful considering the alleged pin configuration of the device.
They didn’t, They altered a existing design just enough to insert their hardware Trojan. It’s not hard to do on a densely populated server board. Its not some toy Arduino .
QA won’t notice as long as it works according to test specs.
As far as the device goes my guess the Feds are leaving out some details. But it’s certainly doable when you have the power of the Chinese government behind the plan.
If you ask me, the story was planted by the Chinese. It’s their version of Reagan’s Star Wars program to make the US spend itself to death defending against a non-existant threat.
B^)
This is basically what the NSA (and ASD/GCHQ/CSEC/PET/DGSE/BND/AISE/AIVD/NIS/CNI/NDB/SID/ISNU) do, I’m not one bit surprised that other countries outside the Five Eyes are doing the exact same.
Yes. They want a slice of the pie too. It is only right to share …
Thanks go to dmbas.es at NSA … who thought that spying on people in this way is a good idea. Now everyone is doing it. Bon appetit.
I have heard that a portion of the delays in bringing a new process node size online at a foundry is the number of black runs they have to do for govt agencies before commodity CPUs etc can be banged out.
Well if said PCB mfgr, and Assembly house are in the same country owned by a secretive government…
“Just because you’re paranoid, doesn’t mean someone isn’t out to get you!”
B^)
For whatever reason an earlier comment of mine was repeated when I clicked reply…
When I worked for a hard drive manufacturer, I was surprised at the high percentage of drives (proprietary info – can’t share numbers) that failed the final testing. It made me wonder if the percentage of fails was much lower and so called “fails” were being routed to another buyer.
Just spit balling here, but could it be used as a disabling device? When it receives a very special signal from the BUS it would trigger a short rendering the board useless or at least damaged? Would be a very effective way to cripple infrastructure.
Very good point.
OK. The Processing is done in the BSC. The “device” Just “controls a few Data liebes”. Think i2c? But tell me which bus coupling “unit” hast more than two contacts?
These are separate components soldered onto a board. I am more interested in the supply chain vulnerabilities of 3rd party Chip Fabs. Imagine, you mail your chip design over to a 3rd party chip fab and trust that they are doing as you ask, but those fabs could essentially build whatever they like into your chips. And detecting the infiltration would be damn near impossible with out going through the silicon level by level. That is what is scary to me…
It is not expensive (compared to making chips) to get a nice layer by layer image of the final produced product. Many companies do this for reverse engineering or IP theft detection. A company that designs chips will likely also have a failure analysis lab that can decap and selectively etch a die.
(hit ‘report comment’ by mistake: blast)
I work for a large chip design company. We build our first run of a chip in our own fab, and when it comes back we have giant plots of the mask design. Of course we can’t see every layer, but we can see the top metal and some details below that. We use those plots for probing during initial debug: under a scope, putting little needle points down into the metal sections of the die to measure bias voltages and look for charge accumulation.
When we get stuff back from our contract fabs, we get wafers back first, and go through the same process, and yes, we have in-house FA that decaps problem chips.
At least for our workflow, it would be very challenging for a nogoodnik to make mask changes and get extra functionality out of our chips.
How often is that checking performed for proven designs? If the nogoodnik waited a year could they make changes without being noticed?
You do that with the first examples of a new design. You certainly don’t do that for every chip that goes through production. An attacker wouldn’t need to modify every chip in the world to carry out a successful attack–they could just pepper a few slightly modified chips in and play the odds until they compromise a system owned by their target. When we’re talking about server farms, those odds are pretty good. They certainly wouldn’t send it in the first batch that will face the highest scrutiny, do you think they’re the three stooges or something? These aren’t script kiddies trying to scam some credit card numbers or some nudes from icloud, it’s the People’s Republic of China.
This is a great example of why grey hat is important. Legit engineers and technicians aren’t used to the strategies used by a professional state-level attacker. You’re used to the procedures of normal, run-of-the-mill QA which was designed to catch careless mistakes, not careful deliberate attacks by a highly competent agent. We live in a slightly different world than the nogoodniks, so we often falsely believe to be sufficiently prepared and know all about the subject. But that’s a dangerous state of mind. Nation-state cyber warfare is a very competitive and fast-paced game. You really need somebody who has personal experience carrying out actual covert attacks. It takes a criminal mind to catch a criminal in this case.
And if we can get THAT person writing for HaD? Oh, boy look out Pulitzer.
I smell bullshit ALL over this.
Anybody remember the Hong Kong treaty with England?
I think it would be best to wait for investigation details instead of making hasty olfactory judgements right at the outset.
While the article is specifically speaking of Amazon and Apple, my concern is around all these “magical network securityy appliance” sellers. I wouldn’t be surprised if many of them used Super Micro the same way as Apple does use Pegatron et al. for the manufacturing of their iPhones. These would be high value targets for sure.
Or those millions of black boxes sold for “never pay again for satellite TV ” …that end up even in the homes of people who should know better …
anybody brave enough to pick up some SMCI stock?
You already would have made around a 30% profit if you bought right at the bottom of the dip when this story came out. Too late now, unless more details come out and it crashes again. But you probably wouldn’t want to buy if this gets confirmed by an independent investigation.
I don’t doubt for a second that they could implement something in a tiny space given how small we can make transistors these days. https://www.extremetech.com/computing/274772-researchers-create-computer-the-size-of-dust
That article stinks of bullshit. Die shots or GTFO
Looking at the animated gif showing the location of the chip on the bloomberg site, the first image shows “AOC-GEH-i4M” on the board. And a quick web search finds the name of that board to be a Supermicro MicroBlade MBI-6128R-T2 ( https://www.supermicro.com/products/microblade/module/mbi-6128r-t2.cfm )
If there is a dump of that board on ebay sooin is probably the board with the bad chip.
I think the Bloomberg piece is more technically naive than bullshit. They write for businesspeople, not hackers. They’re not going to post die shots.
The reporters may not even understand everything — they’re relying on their sources. And if the reporters have a strong enough agenda, maybe they _can_ find 20 sources to tell them what they want to hear.
But BMC vulnerabilities are widespread and well-known. It would not be incredibly inventive to exploit them. Mysterious “Canadian security researcher”, if you’re out there reading Hackaday, give up the goods!
Do you really think they would release exhaustive technical details about an ongoing investigation of a state-level cyberattack, and that if they don’t immediately dump all information it’s BS? That’s terribly naive, and I think it comes from being specialized in electrical engineering and not security, so you get a false sense of expertise on a laterally-related subject. This is a bias to look out for.
The Bloomberg animation shows a model number in the first frame: the Supermicro B1DRI.
https://imgur.com/a/jSpKWsz
There are some on the ‘bay, anyone want to check it out? e.g. item number 202310753506
I’m *this* close to buying this so Hackaday could do proper reporting on the Bloomberg piece. *However*, there’s nothing saying *that particular board* has whatever magical device China implanted on motherboards.
Basically, I’m ready to drop $600 of my own money on doing good reporting, but the null result I would get would be invalidated because ‘it’s not the right board’ or something. Yeah, that ebay auction is a red herring.
I can understand your reluctance to blow $600 on a maybe when all the source we’ve got so far seems to imply specific production runs for specific clients.
I would agree it would be targeted. Makes you wonder who is contracted to recycle the old hardware from Apple and Amazon.
Bloomberg caused a drop in share prices for Super Micro Computer (~ -50%), Amazon (~ -2.2%) and Apple’s (~ -1.8%). I wonder was there any short selling done.
I can’t imagine Apple or Amazon recycling hardware like that in an intact state. My company has a big machine for shredding all hardware with non-volatile memory.
@T_Riddle I expect that for the data storage. But the scrap has to go somewhere, and that is usually outsourced, since it is not typically a core business function.
Any SEC violation would attach to the affected companies for withholding material financial information. Not Bloomberg. Of course I suspect that point is likely moot as the companies are likely under NSLs and have immunity.
Yeah, that’s fair. Thanks for thinking it over.
I’m concerned that you think “proper reporting” would be for a security layman to buy a single random mobo off ebay to try and disprove an attack which would obviously be targeted. You apparently have a bias to disprove this report as well. I say let’s wait for the details to come out and see what it looks like from there. If anything it would be surprising if China didn’t put bugs and rootkits in our stuff when it’s so technically plausible. Y’all have forgotten what the Cold War was like, and technology is wayyy sneakier now than it was then.
Well that explains why the TV had a quick article in it about checking out all Computers and Electronic equipment on our Little Navy Ships for these small chip being implanted onto the motherboards.
No doubt.
It’s the price we pay for off-shoring our electronic industry to save a buck.
Exactly. You were sold out by your own greedy corporations and bribed government. Furthermore, your tax money is being spent on endless wars around the globe, achieving precisely nothing for you, and making the US almost universally hated.
We don’t need to be liked by you.
September 11, 2001 begs to differ
And we don’t need your bombs and coups, thank you.
You’ll get them anyway, because we can.
Spot on!
that dubious news make you want to buy american motherboards, and raise your defiance of china. Get yourself a C64, quick!
Apparently bloomberg made up this story. Apple and Amazon both refute the substance and the details, and said bloomberg got it all wrong.
If Apple and Amazon had another government with full access to their infrastructure for 3+ years, they would say that publicly, wouldn’t they ?
If it is true, it will be verified in a few days if the “Supermicro MicroBlade MBI-6128R-T2” board has a “special” chip on it in the location shown by Bloomberg. The special chip is roughly halfway between the SuperDOM port (SATA 3.0) and what looks like a connection.for a fan.
Quoting the Bloomberg article “Three senior insiders at Apple say that in the summer of 2015, it, too, found malicious chips on Supermicro motherboards. Apple severed ties with Supermicro the following year, for what it described as unrelated reasons.”
If they got three independent senior sources saying the same thing, then it’s hardly “made up”. It can still be totally wrong, of course, but it looks to me like the Bloomberg reporters did their homework. Or at least they’re saying they did.
(I don’t know either way, mind you.)
Hard to evaluate really, on the one had we’ve seen a dozen times or more some top 500 companies denying and denying they’ve been hacked until customer info turns up on the dark web a day or two later. (Two classic “no we weren’ts ” I can think of are target and equifax.) Apple tried denying the recent battery life tuning thing for a few days.
Then also they could be under National Security Letter, with the machines involved internally quarantined and being used by NSA or someone to feed false info to Chinese or something, or just under observation to get information about Chinese aims and intentions.
Then also it would be a highly convenient time with midterms coming up for republican leaning media outlets to put the public into a hating China mindset and thus give credit for a tough on China stance to the Republican incumbents and candidates.
All of this, you are thinking in the right way! just like the kavanaugh was a distraction for all the tax breaks they passed.
AT&T denied that they gave the NSA free reign in their wiring closet in their San Francisco hub. Yet it was done. There are reasons to doubt the story, but Apple and Amazon denying isn’t one of them.
Well, they probably gave the NSA guys badges.
But yeah, AT&T, “Ma Bell” has been in bed with NSA for a long, long time. They saw themselves as a critical part of the nation’s infrastructure (which they were) and as a member of the national security team (which they also were), as they ran AUTOVON and AUTODIN for the government.
Not just AT&T — every telco except QWest, which didn’t do so b/c it was illegal at the time. (When the tapping came to light, legislation was passed making it retroactively legal for the companies to have done so, but that don’t make it _right_, IMO.)
But for a few years, _every_ major telco in the US was breaking the law simply b/c they were asked nicely by a gov’t agency. It seems a bit naive to think that the Chinese gov’t wouldn’t have the same pull on its own firms.
Sure. Like a major corp like Apple and Amazon would openly admit they got hacked by a major board supplier and their sleazy and corrupt Chinese board manufacturer.
It will never happen as it makes the executives look stupid for getting played.
But the fact is a state based player has the resources to make hard trojans a reality, especially when most of the world has it’s electronics manufactured in said state. A state known for it’s hacking of foreign nations and stealing IP al over the world.
Really trusting a company that’s mostly staffed by Chinese nationals to design products used by our banks, military and government and then manufactured in China where everything is for sale is beyond stupid, it’s insanity. All to save a buck
Apple, and nearly all other electronics companies, got scammed in the 90’s with counterfeit electrolytic capacitors. No reason why they couldn’t get scammed with some deliberately malicious component installed by a Chinese PCB manufacturer or board assembler.
They wouldn’t even have to specifically target companies like Apple and Amazon. High end server boards will nearly all end up at companies handling very large amounts of money and/or working on high value technology.
Have the stealth component open a hole through network security so someone outside the company can access the remote management system, then update the RMS code to open access to other parts. Grab data, send it off to some anonymous file upload site then replace the RMS code with the original and nobody will spot the hack – especially if the hackers clean any logs of the suspicious upload. If the hackers are really good they should be able to get rid of any file system notations of access to the logs to edit them.
You’re just trading unquestioning faith in one source for another. It’s standard practice to play down security breaches, and there’s always the possibility that the denials at came from a party that just doesn’t know. The exploit was allegedly found by an outside security firm. It’s not only possible but highly probable that Apple and Amazon haven’t found all the flaws in their systems yet–that’s the definition of a zero day exploit, after all.
Notice that the rebuttals deny that they ever found a bug, not that it could exist. I’m sure their security is currently checking into it regardless of whether it’s real or not. As they should. Credible threats should be taken seriously.
I really don’t see why everyone is so damn impatient to dismiss this reporting as fraud and believe the tech companies’ rebuttal at face value considering the VERY RECENT cases of security breaches being covered up. Remember Equifax? Did they really face any dire consequences for mishandling an attack and misleading the public? Let’s maybe wait and see the details when they come out?
Complete bullshit.. wonder what the true motivation for the Bloomberg article really is? hmm…
Complete bullshit.. wonder what the true motivation for this comment really is? Hmmm…
As if you know what’s really possible in modern cyberwarfare or understand international corporate intrigue. Please.
Short-seller Enhancement Commission?
Are you the same bloke who’s mentioned the SEC and Bloomberg influencing share prices 8 or 9 times, and little else, in the comments? Cos yeah maybe they are, but if they are doing, the journalist in question is going to be in some big shit, and certainly flush his career as a financial writer down the toilet. So unless he’s really shorted the shit out of Supermicro, I’d guess not.
But while we’re throwing crazy theories, you 50 cent party?
So here’s a retort from Amazon, directly:
https://aws.amazon.com/blogs/security/setting-the-record-straight-on-bloomberg-businessweeks-erroneous-article/
I would like more information on how this could be detected whether or not it currently exists. ‘Cause if it didn’t exist before it certainly will now.
I’d ask Bunnie his thoughts but would not want to see him in a dangerous situation if it were true.
Here’s the correct attitude. I have no idea if this specific breach was real or not, but if the attack vector is possible it will be attempted, or more likely it already has. It certainly sounds possible. It would be foolish to ignore it.
So show me just one photo of a board with this thing in place. Just one.
Sorry, that is Classified Information.
B^)
FYI: This is a more technical look at the alleged claim by a 3rd party
https://www.servethehome.com/bloomberg-reports-china-infiltrated-the-supermicro-supply-chain-we-investigate/
I wasn’t worried before because, well, Bloomberg reporting on a technical issue. ‘Nough said.
After reading your link I am much more worried.
“This is where the hacked chips are located on the board that Bloomberg depicts. This also shows that a Supermicro PCB is spun for multiple products.”
The same PCB used for multiple designs means that the only difference between the boards would be in how the board was populated. The only requirement would be to populate a pad on the PCB- hijacking those traces to provide access to whatever you wanted. It could even be a custom chip.
Keep in mind the objective of this would be to monitor traffic to and from the server, not take over the server. Any behavior outside of that would be more likely to be noticed. This sort of action could be done by one actor working alone in a company.
Sadly it is the type of behavior I would expect from a country that has built a business model on IP information theft. (My apologies to the vast majority of people and companies in China that would never consider doing such a thing.)
What sort of things do you look for to determine if this is happening? Duplicate packets sent to different IP addresses?
I don’t think you would find duplicated packets as compression/encryption wouldn’t have much of an overhead. I would think something like a keylogger in the background to look for credentials or interesting packets, remote batch job/interactive session for exploring the machines and their connections etc.
If I were the attacker, I would rent out a machine on AWS in same country as the target machine to better hide my Chinese IP address.
Maybe paired packets sent within a certain latency?? That assumes you have a limited cache in your plugin chip.
There are just so many ways to obscure this that I really am at a loss. Once you get in the silicon you can do anything.
I don’t see how the companies could screen for this without putting the servers in a sandbox and running a ton of parallel tests. The server equivalent of the Voigt-Kampff machine tests.
I honestly think it is funny how some people think open source hardware is a fix for a supply chain problem.
It doesn’t matter if it is open source or closed source, if one’s supply chain is crippled then one’s supply chain is crippled end of story.
A fix is to change the supply chain and have more oversight on it.
Though, I am at the same time somewhat skeptical to that this attack has actually happened, even though there is plenty of room in one of those resistor network chips or similar to carry out an attack like this. And the feasibility of doing an attack in this fashion isn’t impossible, and likely not all that hard, all things considered.
I’m hearing reports that it was a 555 timer.
LOL!
You win!
Stupid China, don’t they realize we can find these with microscopes and x-rays. If they were smart they’d hack those devices to falsely claim nothing was found. But good luck with that, most of those machines are made in….. Ok we may have a problem here.
I’ll believe it when more evidence comes forward. It’s on the line of whether HaD should report this but it’s of interest so I guess it’s on the right side of the line
On the line? I know having opinions about what news should be on what news sites is all the rage these days, but true or not there is plenty to discuss here.
And it’s hacker-relevant, no?
The trick with a physical hack like this is that it leaves hard evidence behind. The NSA uses their TAO division’s hardware resources sparingly.
Relevant both in security, as well as showing state-of-the-art. When even consumer level electronics can do this. Just think what spook level can do. Statistically it will not affect most of us, but then again none of my government “security clearance” leaks have ended up in China for potential blackmail.
How can you trace the code on an external processor that produces the output you are investigating.
@Hackaday (not sure if that works here). I know there is a lot of skepticism in these responses. As of now everyone accused is denying this. I think this is bad reporting (on Bloomberg, you are just citing a trusted-ish source).
Just to recap: Bloomberg calls amazon and goes “there is a supermicro board with a chip next to the resistor panels 1.3 cm below the pcie (or whatever)” and Amazon looks and either goes “yes” or “no.”
They are going No. Everyone is going no.
Can you guys do a follow up this please when there is more information. Also I full heartedly support the above comments that we should pull some Hackaday super-heroing and investigate these boards ourselves. They are supposed to be directly connected to the processor, so it can’t be impossible to track down even with a complicated modern board? If this community can’t pull together to get to the bottom of this… id be disappointed to say the least.
Thanks. Said with all respect for you guys.
For sure! Get us the boards, and we’ll have a poke-around!
I’d love to know if the “chip” is on the I2C bus, for instance.
Have you considered that the exploit may have targeted a few specific boards on select orders? To execute an attack you don’t need a rootkit in every computer in the world, and it would be stupid for the attacker to put it on more boards than necessary–it would only increase the likelihood of detection. You aren’t thinking like a black hat here. It’s not going to be that quick and easy to confirm it happened or it didn’t, and the rebuttals are suspect precisely because they came out so quickly.
These are reassuring words for the stockholders, not credible reports from their infosec team. You shouldn’t believe Bloomberg at face value, but you definitely shouldn’t believe Amazon either.
Not likely. SMT production are highly automated. So at the very minimum, you would have a production batch.of the modified motherboards.
Passive parts wither comes in a reel (e.g. 5000 per reel) or a bulk loader. If the fake parts are simply stand in for the actual part, then you still have to somehow figure out if the current reel have the fake ones or the real one. Easiest is to use the fake reel at the beginning of a batch and remove them when done. Normally the remaining parts goes back to the inventory – could be common parts for everyone or specific to an end customer.
If the PCB are modified to accommodate such a chip, then the pick & place machines have to be programmed for it.
I wouldn’t even mess around at the factory if I was a Chinese sigint bod, I’d just stroll down to the export customs warehouse where they’re gonna sit for a week, show my 007 License to Kill, load the motherboards in a van, drive away, take a leisurely 4 or 5 days to rework them and tuck them back in packaging like nobody was ever there, and take them back again before they’re due to get approved and let out of the country.
There have been published hacks involving modification of HDD firmware .. once you do that, the system is pretty much pwned .. you can read everything, and you inject whatever you want, whenever you want. If an smart enthusiast can do something like this “for fun”, I’m sure state sponsored actors have weaponized variants of this sort of stuff honed to perfection.
http://spritesmods.com/?art=hddhack&page=3
No, no…..
That’s a picture of a diode.
I realize that hack a day is the site where the zener regulator is derived, but still it’s just a diode.
What picture? The one with the article is just for illustrative purposes, it’s not the actual component.
And it’s not gonna be a menacing package with the words “malicous rootkit” stamped on it, now is it? If they were trying to hide it on a circuit board, don’t you think they’d disguise it as a legit component and have it mimic its functions?
Derided
supermicro’s press release: https://www.supermicro.com/newsroom/pressreleases/2018/press181004_Bloomberg.cfm
>In an article today, it is alleged that Supermicro motherboards sold to certain customers contained malicious chips on its motherboards in 2015. Supermicro has never found any malicious chips, nor been informed by any customer that such chips have been found.
>Each company mentioned in the article (Supermicro, Apple, Amazon and Elemental) has issued strong statements denying the claims:
>Supermicro has never been contacted by any government agencies either domestic or foreign regarding the alleged claims.
Why would the Chinese would add a separate “malicious chip” on the PCB that’s so easily detected when they can disguise the same logic on the silicon of any other chip? Far more likely the ESP8266, Atmel, Allwinner, Rockchip, etc have embedded malicious silicon. Everything cheap from China comes at a price…
I don’t think China is all that interested in disabling, or monitoring, all the cheap electronics in the world. Expensive electronics is more likely to have interesting stuff on it, or have more ‘interesting’ results if disabled.
The Register has a good write-up of this story as well, and creates the same response, don’t know what really happened but something happened.
Lots of reasons. A single house doesn’t produce every component. If they put it upstream in the supply line, say the CPU manufacturer, it would be harder to guide the compromised systems to their target. Putting it in at assembly gives them control of which order gets which units.
If they just put it in every chip made by a widespread manufacturer, they only vastly increase the chances of detection for no real benefit. And what if the specs changed and the client decided they need a different BCM at the last minute? No problem, it will still need some passives near the firmware memory…
Not saying there aren’t exploits out there right on the CPU die. That’s probably been done as well. You should see some of the crazy stuff the Soviets hid in typewriters back in the 70s. It’s only gotten easier today. Anyone saying it could never happen is inviting hubris.
Seem like a whole lot of work, and would take some time to figure out where to place the chip, in such a way as to not effect normal operation of the board. It’s a physical alteration, so if something strange is going on, somebody is bound to go looking for the source, eventually spotting the mystery component. There is a trail, it would be know where the board was made, where the components were purchased. Not really sneaky, for the professionals. Seems like traditional cyber attacks are still very successful, why bother with the hardware, and leave such an undeniable trail? I wouldn’t think you’d need to supply a manufacturer with a complete schematic, full documentation of the board, just the traces, and parts layouts. It’s not the manufacturers responsibility to check your work, correct your mistakes. It takes a lot of time to reverse engineer something, and alter it flawlessly to function as the original, with a new feature integrated. After the work is submitted, there is a certain expected wait time, before the boards start shipping. Thing that added time it would take to reverse engineer and place the part, flawlessly, would be noticed. How long can it be expect, that these boards will be in service, before being replaced by something better?
Seems like this is mostly speculation, and with most stories based on “unnamed” sources, probably had a different intent, than to point out the issue in the headline. Did anyone actually pull the chip, test it, take it apart, to see what it actually could do? Could it have been a substitute part, for one that wasn’t available, or a quick, last minute revision, but nobody bothered to update the rest of the team that there was a change, from what was on the “final” plans.
It’s kind of like when Pres. Obama declared that the Russians hack the 2016 election, slapped them with sanction. Two years later, it’s still being investigated, we were assured that their efforts had no impact on the election, but no solid proof of collusion. Clintons were more Rusia friendly, would think that would have been their preference. There very well could have been Russian hacking, specially the Email phishing, common activity, but not really a government thing, just average thieves, or curious kids, who didn’t care about politics.
Still think it’s mostly speculation, a mystery part, and a lot of maybe’s and fantasies. Lots of stuff can be done, but is it really worth the time and resources needed to pull it off? Sure, there all kinds of senseless projects showcased here, took a lot of time and money to put together, just because they could, or it was a good build adventure, but the final product is neither cheaper, better, than what could be bought, sometimes the finish project doesn’t really do much of value at all. It was done, as a proof, just an idea, and made it work. Installing a malicious chip is risky, and no guarantee of any return. The source of the alteration would be easily fingered, and held accountable.
A bit suspicious of the reality of this, as a lot of people here.
Still, when examining Bloomberg’s photo from the original “The Big Hack” article, if this is the real thing, we can see that the purported malicious 0603 or alike chip is located *on* the footprint of a non-populated TSSOP-8, quite possibly an SPI EEPROM. This could be an “optional” memory IC which would be overridden here to store some parameter for the malicious code. But to me it is obvious it cannot contain all what is needed for a real hack: the real hack payload *must be already contained* in the BMC code. The action of the malicious function is only *triggered* by the presence of this memory, which could e.g. contain the target hostname to contact, to avoid putting suspicious hostname or IP address in the BMC code itself.
So, it would be interesting to reverse-engineer some Supermicro’s BMC code to see if it is probing for something special on the SPI (or maybe I²C) bus.
Can you imagine, at the height of the cold war, that the USA would have allowed companies to source production of computers to Soviet block based production lines to improve the share price of those publically listed companies.
Out sourcing to China was always going to go this way. It’s what the Chinese planned from the start of the entire exercise.
They continue to beat the western world into submission by using western ideals of corporate greed and the accumulation of money as weapons against us.
Aquisition of money is just another resource the Chinese use to wage war, unlike us it is not th eultimate goal.
They are winning and will continue to do so globally until we stop handing them the planet on a plate through our own stupidity.
It is not computers, but US did use titanium from Soviet Union to build spy plane during the cold war. You wouldn’t want Russian electronics anyways because they were not as advanced as the west.
https://en.wikipedia.org/wiki/Lockheed_A-12
>In his book Skunk Works: A Personal Memoir of My Years at Lockheed, Ben Rich stated, “Our supplier, Titanium Metals Corporation, had only limited reserves of the precious alloy, so the CIA conducted a worldwide search and using third parties and dummy companies, managed to unobtrusively purchase the base metal from one of the world’s leading exporters – the Soviet Union. The Russians never had an inkling of how they were actually contributing to the creation of the airplane being rushed into construction to spy on their homeland.”
At my first Cyber Security Summit about ten years ago. I warned them of such an attack. This is exactly what happens when companies want better profits at reduced cost.
So you are saying ‘Design, manufacture, code and maintain all in the USA’? That is a nice idea perhaps but that time is gone, and went about a hundred years ago.
Against industrial espionage and sabotage it’s no defense either.
“but that time is gone”
Only because of greed. There is no reason whatsoever it cannot be restored.
“and went about a hundred years ago”
Hyperbolic much? As a superpower competitor, China is VASTLY more dangerous that the Soviet Union ever was due to Chicom capitalism and the greed induced stupidity of US manufacturers handing over IP to gain access to a market which they will never be allowed into in a SUSTAINED basis – China will simply use the willingly provided or stolen IP to set up their own industries.
Considering these FACTS, how INCREDIBLY F’ING STUPID would it have been considered to be to allow the Soviet Union to manufacture network hardware for US use had that sort of thing existed then?
I was thinking more about the rise of the USA as seen from Europe.
False dichotomy. There are plenty of other possible countermeasures. Splitting up the sourcing of the components instead of exclusively using a single foreign country with proven motivation to launch cyberattacks on the US would help a lot, not that every single step would have to happen here domestically. And we could do better on the software side. If we did lots of hardware-level encryption and flashed all the firmware here in our country it would certainly make attacks harder.
I firmly believe that 100% unhackable, perfectly secure hardware is not possible, much like a 100% efficient engine. And even if it was possible, nobody would want to pay for it. But there are other precautions we should be taking right now and it’s going to bite us eventually.
I can understand the appeal of out-sourcing production to the cheapest manufacturer, business is about profit, usually driven by greed. I can’t understand why our government, or any company, would which we are force to trust with our safety and security, would send production of equipment, which ultimately would be tasked with keeping our most valued information, finances safe and secure from thieves and enemies. Personally, there should be some accountability, whether or not there is any truth in this story. Data security should be a top priority, where our government secrets are concerned, should never be trusted to another country. I’m pretty sure we have the means to make the same boards here at home, in a more secure facility, the higher price tag would still be worth the confidence that we are doing everything possible to protect our data for foreign eyes. We have enough domestic thieves to worry about, need to shut the door on everyone else.
I’ve gotten a few letters over the years, after shopping at one store or another, about a security breach, and my debit card/ personal information may have be stolen. So, I get a new card, and a free offer for a year of credit monitoring, usually through Equifax. I’ve never taken them up on the free offer. Mostly, it’s handing over personal information over to someone, who doesn’t really need it. Also, Equifax got breach, least once that made public, also seem like a prime place for an employee to make some good money on the side. I’m not too worried about identity theft, haven’t used credit since I paid off my house, and all other debts. Been debt free for over ten years, no credit activity, very low credit score. My bank kept sending me card offers, figured a backup piece of plastic would be a good idea, since the debit card was useless for about a week, when there’s been a breach. Bank would issue a credit card, and I’ve been with them for almost 30 years…
Out-sourcing our safety and security, to a know hostile country, should be a very high crime, and should be prosecuted. Never happen, since many of those people hold high office in our government, and have most of their lives. Certainly, they would prosecute themselves, or authorize any poking around, that could harm their financial interests.
“I can understand the appeal of out-sourcing production to the cheapest manufacturer, business is about profit, usually driven by greed. I can’t understand why our government, or any company, would which we are force to trust with our safety and security, would send production of equipment, which ultimately would be tasked with keeping our most valued information, finances safe and secure from thieves and enemies.”
You answered your question in the second sentence with your first sentence. This blogger also knows:
It’s Not Just The Hacking: It’s The LIES
https://market-ticker.org/akcs-www?post=234281
Excerpt:
From the Bloomberg column:
“Over the decades, the security of the supply chain became an article of faith despite repeated warnings by Western officials. A belief formed that China was unlikely to jeopardize its position as workshop to the world by letting its spies meddle in its factories. That left the decision about where to build commercial systems resting largely on where capacity was greatest and cheapest. ‘You end up with a classic Satan’s bargain,’ one former U.S. official says. ‘You can have less supply than you want and guarantee it’s secure, or you can have the supply you need, but there will be risk. Every organization has accepted the second proposition.'”
As I have repeatedly warned over the last decade in this column that first article of faith is complete crap. The PLA is the government and business both in China. To believe that they wouldn’t “jeopardize” their position is also crap for the simple reason that the entire premise of offshoring is to seek the cheapest price risks be damned as nobody in the United States in corporate or government work is ever prosecuted for screwing their customers.
Let me repeat that: NOBODY EVER GOES TO FEDERAL POUND-YOU-UP-THE-*** PRISON SO LONG AS YOU ARE A LARGE COMPANY OR GOVERNMENT AGENCY.
EVER.
There’s a lot of comments on this article, but. Hackaday should be more diligent and it’s “Featured”. This article is supposition. There are no facts in it. Is it possible? Yes, I know for a fact what governments are capable of. And there are plenty of news stories depicting that router x has a backdoor for government Y and so forth, which are very specific, why isn’t this article? But to post a story about this without any factual evidence is damaging to the IT industry. So IT person (me for a few hours) freaks out that they need to replace all of their hardware. From my reading up on this, the likely story is that there is a security flaw in the BMC as stated by Amazon when they acquired supermicro. This is a bug, not some government spy hole. Now, there maybe some server manufacturer that is using supermicro motherboards and building complete servers with this “chip” backdoor on them. but single them out if this is the case. Otherwise It’s all B.S. And “oh it’s possible” as an IT person you account for the possible. But if there is hard evidence that some equipment or software is compromised, then it needs to be replaced.