The first clue was that a number of locomotives started malfunctioning with exactly 1,000,000 km on the odometer. And when the company with the contract for servicing them couldn’t figure out why, they typed “Polish hackers” into a search engine, and found our heroes [Redford], [q3k], and [MrTick]. What follows is a story of industrial skullduggery, CAN bus sniffing, obscure reverse engineering, and heavy rolling stock, and a fantastically entertaining talk.
Cutting straight to the punchline, the manufacturer of the engines in question apparently also makes a lot of money on the service contracts, and included logic bombs in the firmware that would ensure that revenue stream while thwarting independent repair shops. They also included “cheat codes” that simply unlocked the conditions, which the Polish hackers uncovered as well. Perhaps the most blatant evidence of malfeasance, though, was that there were actually checks in some versions of the firmware that geofenced out the competitors’ repair shops.
We shouldn’t spoil too much more of the talk, and there’s active investigation and legal action pending, but the smoking guns are incredibly smoky. The theme of this year’s Chaos Communication Congress is “Unlocked”, and you couldn’t ask for a better demonstration of why it’s absolutely in the public interest that hackers gotta hack. Of course, [Daniel Lange] and [Felix Domke]’s reverse engineering of the VW Dieselgate ECU shenanigans, another all-time favorite, also comes to mind.
Think the point you are missing in spinning it this way is that if the workshop doesn’t know the hardware well enough to be able to enable them already, then it doesn’t know it well enough to service it properly. Disabling safety measures is not an underdog hero story
Would you extend this to your car? “Ok, the manufacturer has written a service manual, but you must also know secret unlock codes or you are not good enough mechanic to change the oil!”
And claiming any safety benefit from bricking trains that just sit at a service yard for 2 weeks makes no sense.
Given how little I know about car maintenance, yup. If they give me adequate warning then put it into limp mode so I can get it to a garage to get the timing belt or whatever replaced – though I’d like my choice of garages, not just dealerships. I think BMW do this though they restrict to dealerships.
But we already have a basic version of this this mandated by law in annual MOT tests. If it’s not been checked by a qualified technician in the last 12m, the police will pull you over when you register on ANPR. Keeps some of the worst cars off the road at least.
“But we already have a basic version of this this mandated by law”
Are the inspection conditions mandated by the *manufacturer* or mandated by *law*? I’d guess it’s the latter.
This isn’t that. This is a manufacturer literally putting in “automatic fail” into an inspection because it detected that it was being serviced by a competitor.
Have you ever tried to change the oil, or for that matter any sort of
mechanical work on a Lamborghini Aventador ? or Ferrari ?.
The amount of labor involved (and requisite specialized knowledge)
is very involved.
A friend paid close to $8000 for oil and spark plug changes on his
2018 Aventador (granted some of it is the “Lamborghini tax”, but the
design is very complicated and does not lend itself to easy service).
I don’t think you’d want to take a $400,000 car to Jiffy Lube !
Heck, even the Corvette C6 Z06’s dry sump system is too complicated
for mechanics unfamiliar with it’s unique design.
Supercars are not comparable to commuter trains, which are meant to be serviceable by the lowest bidder – and such clauses are typically included in the contract to manufacture them.
I’m sorry an Aventador is nothing special changing oil is the most basic of tasks after changing a wheel. Changing 12 spark plugs may be harder due to access but it’s no more than half a days labor. $8000 is a 90% Lambo tax and it cannot be justified.
I don’t know a lot about supercar-level luxury crap, but at least when you look at the “standard” luxury brands, if you compare Lexus/Toyota or Acura/Honda, part of the reason the service costs are higher when done by the “luxury” dealer (as opposed to the base dealer) is a combination of higher customer service costs and increased insurance.
Obviously changing oil is an incredibly basic task, but go talk to any mechanic and they’ll tell you stories of someone starting out who forgot to put the oil cap back on when the car was returned to the customer: and the cost to the dealer is a lot different between a $25K car and a $1M car.
So yeah, it’s definitely not due to the labor. It’s due to the brand.
What are you talking about? The other shops knew the hardware perfectly well. They work on trains all the time, it’s what they do. What they didn’t know was the software, which was closed-source, obfuscated, and deliberately sabotaged by its creators to try to make normal service techniques not work. Which is, and I’m being very generous here, at best fraud.
youre so wrong it bothers me to all hell. the “hardware” is a progammable logic controller. a computer.
the manufacturer installed new firmware with added (also undocumented) traps, in some cases days before the trains were scheduled to go to competing shops for service. without documenting the firmware version change.
the traps disabled motors but reported no faults. everything else has a human readable fault codes, this code just says “all good” and keeps the motors turned off without reporting a thing.
most of these conditions were literally “has the train been off for over 10 (or later 21) days?” “are we currently within gps coordinates of one of these 4 competitors railyards” or “has the serial number of one of several non programmable and not safety critical critical components changed from last time” aka parts swapped out for the same official device bought from the same manufacturer. not safety, just screwing the competition over.
So yea, if the customer had kept the trains off for two weeks they also would have seized to function, without anyone touching a single button. that would never happen because they have to run 20h/day to be profitable (according to the ccc talk).
The contracted company actually took the trains apart according to service manual, sent the parts TO THE MANUFACTURER for overhaul as specified, put them back together according to service manual and the computer refused to work, without any fault indication or code.
the hackers got their first train moving less than an hour before the customer arrived to take the trains back to the (more expensive) manufacturer for service and noone would have been the wiser.
Uhhh hardware service != The same as modifying software.
Any monkey with a wrench/tools, the time and a manual can fix/service hardware correctly.
You are acting like like a simple hardware service center should be able to reverse engineer boobytrapped code.
Nah bud.
The point you are missing is that the competing workshops are not some “Fred’s Engines” garage shops in an alley. They are fully qualified and some of them they belong to competing train manufacturers. Provided with necessary documentation they are fully capable of conducting repairs (and they know, if they aren’t).
“Safety measures”? I may have been born yesterday, but I was up late last night. When I was a kid it was politically correct to make “Polish jokes”. It sounds like you’re trying to make a joke about “Polish safety measures”. I can’t sign off on that. That’s bad juju.
“Think the point you are missing”
I think the point you are missing is that you should’ve watched the talk. The original purchase contract with Newag *required them to give all of the information required to service the trains*.
If a manufacturer wants to say “this train must only ever be serviced by us” and that’s accepted by the customer, that’s fine. Well, no, it’s not *fine*, but it could conceivably be legal in some places.
This is not what happened.
“Disabling safety measures”
You really, really should’ve watched the talk. There were no safety measures disabled. “Never start if you stop for a while within this lat/long range” is not a safety measure. They even explicitly avoided changing any code at all – they just called functions in the software that were already there.
Seriously, you shouldn’t’ve commented.
>The original purchase contract with Newag *required them to give all of the information required to service the trains*.
Which is why it must have been an insider deal. The purchaser and the company were colluding against the third party service provider. ,Since the customer was a public entity, they could not justify giving the contract only to Newag without considering the competition, so they needed an “excuse” to why the trains can only be maintained by them.
These kind of tricks won’t be pulled by one side only. A company that tries it would take too great a risk for complaints and investigation. What they didn’t account for was that the third party service company would hire the hackers.
I think I read on q3k mastodon that other NEWAG customers started contacting them thru the year after first unlock success.
3 letter agencies got involved they will dig this one out but i think nope. They could have just write tender without documentation requirement so NEWAG would be only one to be able to service those trains.
Was this comment paid for by the train manufacturer?
In this case those “safety measures” were saving NEWAG profits and nothing else. Plus there are EU regulations forcing train manufacturers to provide Full train maintenance documentation to third parties, not documenting something is another NEWAG crime.
No safety measures were disabled – read the story fully. Dragon Sector explained that already.
Wrong. One of parameters of contract was that producer – NEWAG had to provide full repair documentation that allows KD to service their trains at other companies.
https://hackaday.com/2023/12/14/polish-train-manufacturer-threatens-hackers-who-unbricked-their-trains/
https://hackaday.com/2023/12/06/the-deere-disease-spreads-to-trains/
While this topic has been posted previously, what was linked today was new info shared at a conference.
Good point. Thanks.
We are talking about trains! Not about the printer at your home!
If anything goes wrong and a train is involved in an accident, causes by a malfunction, the manufacturer is liable.
Therefore ist must be his priority to prevent service by an unknown workshop.
As long as there have been trains, there have been people who operate and maintain train engines on behalf of their owners, the so-called “engineers”. Are you saying that these “engineers” should not be allowed to ply their trade simply because the manufacturer doesn’t trust their customers to use their product safely?
Perhaps you are right. Certainly the manufacturer should not be compelled to employ people-who-know-the-law-inside-and-out, or “lawyers” as I’ve been taking to calling them, to write binding contracts that insure that the manufacturer is free of liability if these pesky “engineers” cause damages due to their incompetence.
Clearly the best way to prevent undue legal encumbrances is to weld heavy steel plates over anything in the train that the pesky “engineers” may be tempted to get their grubby mitts on and allow access to the go-faster, go-slower, and toot-toot levers, as well as load manufacturer-approved fuel under the make-water-get-hot tank. Only official service personnel may undo those welds and maintain the train.
The “lawyers” are always complaining that this falls afoul of anti-competitive laws as it gives the manufacturer a monopoly on the service of their products. Surely manufacturers are doing this because they’re concerned for the safety of their customers and not to create a new and lucrative revenue stream by artificially becoming the sole source of maintenance for the life of the product. This sort of incoherent nonsense is why they should be ignoring the “lawyers” in the first place. Serves them right.
Sadly the manufacturer has to deal with reality.
If an accident looks like a technical problem he gets sued. Because there is a law that makes him liable.
The he has to prove that it wasn’t his fault, if that’s not possible he may go bankrupt in the process.
And by the way if someone changes the software on a safety relevant PLC, the certification is voided and it is illegal to operate that train.
“And by the way if someone changes the software on a safety relevant PLC, the certification is voided and it is illegal to operate that train.”
Weird, because the manufacturer changed the software on that frequently and there’s no mention of that in any certification.
It’s almost like that they use that requirement as a weapon against others rather than something to be taken seriously.
Uh no the manufacturer is not “always liable.”
In the event of an incident they figure out the cause of the failure, look at the service records etc. If it’s a design issue it’s the fault of whoever designed and tested the part (mfr usually). If it was inadequate service, it’s the service center.
If the mfr find a design flaw they put out a notice for repair which can be done by any service center.
True, but the bad PR and suspended contracts could still kill a company. Not saying that’s the case here, but it is in some places.
So we have two competing concerns here:
1) The manufacturer is concerned that improper servicing might reflect poorly on their public image
2) The customer wants to be able to fully own and fix the thing that they bought
Are these of equal importance? As a society, which of these do you think we should weigh highest?
The second.
The first is actually insane. It doesn’t work. The only way you make a product that can be serviced correctly is by making it easy to service. Which means you can’t lock down *where* it’s serviced. If you make it hard to be serviced anywhere else, the product, at best, gets a reputation for being luxury, but fragile.
Think about it. How would you make a product hard to service? Let’s say you use the hidden fasteners trick, or glue things shut. Great. Now when it does need to be serviced, there’s a huge chance of breaking something, because it’s finicky to open. Or let’s say that you’ve got “special software” that trips a security feature or something if it’s accessed wrong. And now when *your* service centers screw up (and they will!) the product will just ‘accidentally’ break due to your (fake) security feature.
That even happened here! The fake lockouts the company put in place *accidentally tripped* several times, and the trains had to be sent in for service even though *nothing was wrong.*
If a Polish Airlines mechanic does work on a Boeing 737 and screws things up and that results in an accident, Boeing isn’t liable. Why would it be any different in this instance?
In this case the customer put out one contract buy trains(which Newag won) and a separate one to service them(which Newag bid for; but did not win).
These things were explicitly sold as being serviceable by 3rd parties of reasonable competence; including service manuals that were supposed to describe the behavior and required procedures for servicing them.
It might be legal to sell trains that are explicitly only serviceable by the manufacturer(though, obviously, the price you could get for them would presumably reflect the customer’s expectations of being taken to the cleaners on sole-source maintenance for the life of the equipment); and I presume that it’s illegal for the train operators to just call in random shade tree mechanics to wrench around for a bit and call it good; but that’s not the situation here; and what seems like the part that is really damning even in absence of any specific right-to-repair requirements:
Even if you have no general obligation to make your equipment serviceable by 3rd parties; you incur one the moment you represent it as being so when trying to sell it.
The contract was clear NEWAG(producer) has to hand out full repair documentation to KD (Lower Silesia Railways – train owner) so repairs can be done by either KD shops (in case of minor fixes) or third party shops (main repairs). This is a breach of contract.
@ Manfred, do you work for Newag?
https://www.railjournal.com/fleet/newag-comes-out-fighting-in-claims-over-foul-play/
there are some claims/statements from Newag in there ^^ that might mean it’s not as simple as them doing the “John Deere”.
(was linked to in https://en.wikipedia.org/wiki/Newag?useskin=vector#Controversy and I just thought it relevant enough to post it here)
Lol Newag’s claims are hilarious “You can’t prove we wrote that software even though we push the updates”.
Another: “The service company did it because they can’t service the trains”….. But somehow are capable enough to reverse and rewrite the closed source complex software 😂.
Can’t make this stuff up.
I’d be in full panic mode too, if a customer found this out and published the results. This might be enough to sink the company directly, or indirectly as they can no longer exist due to reputation damage or even just having less to do as repairs are no longer artificially inflated.
And rightfully so, I’d say. This kind of criminal deception only serves one company, while disrupting the competition unfairly and saddling customers and their customers, the consumer, with extra costs.
I would expect that top management at Newag are packing their golden parachutes even as we discuss this. The kind of people who are capable of sticking this kind of code into the system aren’t wealthy themselves, and eventually the government is going to find someone at Newag who knows the story and is willing to spill the beans if only to keep from going to jail themselves.
Yeah, they addressed them in the talk – and some of them are worse than hilarious: they’re outright lies. Like, saying that the service company needed to have this software to service the trains?
Yeah, that’d explicitly be against the purchase contract, which required them to give all of the information/documentation needed to service the trains.
Jeez. They couldn’t service the trains so got some Polish hackers involved who reverse-engineered portions of the firmware and found the code that Newag had written to brick the trains if they were parked at certain GPS locations for more than a few days.
The hackers then found some other code that used a sequence of keypresses on the train’s console that would magically unbrick the train!
No safety risks, it was all code the OEM had deliberately put in there to fsck over their customers so they would have to come back to them for servicing. Utterly disgraceful behaviour.
It can’t have done Newag’s reputation any good.
Sad that the S/W engineer will certainly be the one thrown under the bus (train?) for writing the code. When you have a family to support, mortgage bills to pay – yeah right your engineer ethics are to quit your job, walk out. Newag executives, CEO that ordered him to do it – will receive no punishment.
Refer to VW dieselgate, Boeing 737 Max scandal and see that shady code still allows them to play golf, eat caviar.
This scandal also has Polish politicians strangely quiet and doing nothing about legislating against it. I suppose they are getting kickbacks, maybe bribes. Something stinks badly here at many levels.
I dont know about Boeing but In the VW case the big guys actually went to jail , and in this case there is no reason to believe that the softwareengineer will be held responsible ,unless he has benefittet from this feature unknown to Newag .
And rightly so. S/W engineers are among few professions in these whereabouts who don’t need to worry too much. Getting a new job isn’t a major problem for us. Large companies train their employees to watch out for anti-competitive practices like these and report them rather than take part in them because they may harm the companies. Having said that, I hope Newag CEO and execs will get their asses sued off and will be let go in socks.
There was election in October and the new government was established just over two weeks ago. This story went public on December 5th. A public hearing will be held soon in the parliament where the guys from DragonSector will give a talk. I’d say it isn’t going bad.
I’ve only just noticed the bottom line of text in the header image, it made me laugh :)
Writing 0xff to nvram sounds simply like “reset to factory defaults” :)