A week ago we covered the story of a Polish train manufacturer who was caught using software to brick their products after they had been repaired by in independent railway workshop. Now 404 Media has a follow-up story with more information, including the news that the hackers responsible for the discovery are now being threatened by the manufacturer.
The more we learn about this story the more interesting it becomes, as the Newag trains in question began failing after service as far back as 2021. In desperation after services were affected by the number of non-functional units, an employee searched online for Polish hackers and found a group called Dragon Sector. The group was able to find the issue, and are now being threatened with legal action by the manufacturer, who are citing possible safety issues.
It’s clear from where we are standing that Newag have been caught red-handed in some extremely dubious practices, and seem to have little sense of how their actions might not be the best in terms of protecting their reputation. We are guessing that the European regulators will become very interested in this case, and that meanwhile the order books of a company which puts DRM in its trains will start to look very empty indeed. You can catch our original coverage as the story broke, here.
Thanks [JohnU] for the tip.
After reading the previous and current article about the Newag trains, I must make a note “not to buy any of their trains”. Because, even bad publicity is good publicity. Meaning that if I have enough room and money for a train, which should be in about 30 years (according to my local fortune teller, I should have won the lottery by then), I perhaps might buy one or maybe two. Because the real problem is that by then I don’t remember the details but do remember the name.
Anyway, now I think of it… my local fortune teller also has some nasty rumors going round. Don’t recall if they’re good or bad. I’ll guess I’ll find out in about 30 years. Although… I do remember that when I visited the fortune teller and knocked on the door the reply was “who’s there”, which is a pretty strange question for a person claiming to know everything in advance.
+1 …. consumers should vote with their wallets and stop buying those trains.
I know I was on the fence about which train I planned on buying, but now I can cross one manufacturer off of the list.
Same here: my trains were getting old, so was looking to change both my train and my wife’s, and was considering a Newag with custom paint, but now I’ll be looking elsewhere.
I never have and never will purchase a Newag train. They are on my naughty list.
Going to be interesting to see the reaction to the first car maker who bricks their cars.
Like, Tesla?
If you re-sell the car, they disable features the original owner paid for.
This is false. The only features you can buy for a Tesla are Enhanced Autopilot, Full Self Driving and Acceleration Boost (the latter only on some models), all of which are transferrable with the car when sold to a new owner.
That’s wrong, there’s a lot of other DLCs for Teslas. Like the heated seats, that’s an additional fee for the activation
That’s what they did disable. The customers complained, went to court, and Tesla backed off and re-enabled the features.
You my be able *pay for* full self driving, but that money won’t actually make the car drive itself.
That’s simple – the resale price drops.
I really wish I read this article 3 weeks ago. Newag’s contract cancellation fee is horrendous!
So this is basically saying, only we can repair or service our trains. From a safety standpoint, I can understand this. Having someone else doing something to the train that the manufacturer doesn’t know about can open up a can of worms. Well, the train crashed because a third party modified our code without our knowledge.
That’s a good case for closed source and I get it. The other side of the coin is, you have open source where everyone knows what’s in the box, and all modifications are openly visible to everyone.
Yes, opposite ends of the spectrum. Any changes to software that isn’t open source, just like repairs etc. to a train don’t go through the testing the original manufacturer. Back in the day, machines like tractors (John Deere) didn’t rely on software. It was a mechanical machine. A part broke you got another one and you fixed it.. The manufacturers are correct the intellectual property belongs to them, but threatening someone who fixes a flaw in your design making your product perform better? Wrong attitude in my book.
The thing is, it’s not about safety. Even without electronics being involved, independent repair can fit knockoff parts. If something goes wrong, we’ll look at who’s to blame for this fault. If the device you repair is in danger of causing a lot of harm in case of a failure (think: aircraft, trains, medical equipment, etc.), we require certifications for spare parts and repair procedures. Thus, independent repair is possible with certified parts and certified labour, resulting in safety that’s equivalent to the safety afforded by the manufacturer’s own repair.
That’s why there’s no point in having closed-source software either: The device gets certified and spare parts have to get certified as well. Making this complicated just results in one single piece of very old software being the only one that’s certified, opening doors wide to exploitation of security flaws.
So what does certification really afford? Pretty much nothing, just the ability for authorities to quickly remove non-compliant devices. The hope is that certification for alternative spare parts is cheap enough (and possible enough) that it will be done, thus creating a market for spare parts and labour.
If an expensive manufactured product cannot be safely repaired by any reasonably competent personnel with access to certified spares… that’s a big reason not to buy that product.
It’s probably about accountability. If the train has a safety issue, they’ll get blamed and lose orders even if three years later the investigation finds it was a bodge unauthorised part which caused the issue. Assuming the company hasn’t gone bust, and the duff part wasn’t destroyed; and if it’s a firmware issue they can get the bad code off the device and show it wasn’t original.
That said, there’s probably better ways to handle this. What does aviation do?
It’s not like they had random dudes overhauling their trains. It was a contracted company that beat out Newag in the bidding process, but Newag built-in a GPS locator that would brick they trains if they were at a certain location (a competitor’s yard) for more than 10 days, like they would be during an overhaul. This occurred whether they worked on the train or not, which was discovered not through hacking but through them not touching other trains while trying to figure out why the first one they overhauled didn’t work.
The hackers decompiled the code for the train OS(es) and found all this and more in there, as well as found undocumented cellular modems set up for two-way communication, and a special input combo that unbricked the trains. Newag then updated the firmware to disable the unbricking combo.
They have absolutely zero legs to stand on.
IIWIC (If I were in charge), I would be lending a train to the 3rd party repair shop with the following instructions:
“Replace all the software on all the PLCs and controllers with new software. Reuse any hardware you reasonably can, but replace that which you cannot. We will pay for the development, but we own the rights to the end product.
Once that is complete, provide a quote to retrofit all remaining trains with the new software. We will then tell Newag that their services will no longer be needed.”
The problem with trains is that it is a very failsafe thinking world. Which is good, you want to be 100% sure your brakes apply when the brake lever is pulled. Which is why software must be certified failsafe.
But it also means that software for heating and ventilation and things like that (think door LED colours or PA Systems) also needs certifications.
Writing certifyable failsafe software for something as complex as a modern train is not something everyone can do, especially not if you’re not the one designing the trIn.
A guy I know works with railway switching gear, and according to him there’s no real fail-safe certifications as such other than “this is how it’s been working for the past 100 years”. There’s hardware that is built -exactly- like it was back in the day, and more modern technology is simply layered on top with each layer becoming part of the standard as it gets “tested by time”, aka. nothing bad happens for long enough time, so it becomes the way things are done.
in reply to Dude (speaking for germany here): railway switching (actually the whole railway safety ecosystem) has had about 100 years of careful debugging when they started to use relays for safety features. So they decided to do it the same way it was, knowing the amount of blood, sweat and tears that went into the debugging. Now, another 50 years later, this knowledge seems to have disappeared among the folks that replace the relays by computers.
The first problem emerging is the problem with repairs: it takes months to years now to fix things that used to be done on a weekend and two nightly … Sperrpausen (some hours overnight with no trains), and the rules are becoming bloated and unworkable. The computers in this case have no advantage over the relays except being “digital” and “modern”.
Matthias: my understanding of safety certification is that you have defined tests and criteria that the equipment must meet to be certified safe. Doing things by tradition does not really qualify for the same point since there is no definition of what “safe” means. There might still be corner cases where the equipment fails spectacularly, that nobody ever considered, because that case hasn’t yet happened and nobody ever thought to ask “what if this goes wrong, what happens then?”.
Dude: to be considered safe you had to prove that at least two independent faults must happen at the same time for an accident, and every single fault must cause a fallback to a safe state, and except for very restricted cases (tests to verify the criteria in real environment) the new thing must be at least as safe as the safest existing one. They had to change that to be able to certify the computers. At this point it all started to go south.
The 150 years of debugging before that went into cutting the spectacular corner cases nobody has thought of, but now they are back on track :-(
You’d build your own trains? Seriously?
The bricking mechanism Newag uses has nothing to do with any repair. If train is parked at certain locations for more than specific number of hours/days, it stops working. The list of locations covered both existing non-Newag repair shops, but also those that were only planned or under construction. Newag wanted to force PKP (Polish State Railways) to pay quite a lot for repair contracts, so anyone else, who could do the job cheaper was bad for their business. These third-party repair shops must follow strict regulations, so they can’t cut corners and use shoddy parts…
Yes and no. The problem here is that they did not disclose or stated that “nobody but us can service the trains”. That is something you need to disclose in a contract, and it may be a factor when purchasing trains from that company.
Instead of that they were happily letting the companies think the servicers were incompetent by faking error messages!
You can service the trains by a third party – they just won’t work afterwards.
>they were happily letting the companies think the servicers were incompetent by faking error messages!
It’s such a transparent excuse that there’s no way the rail operator wouldn’t suspect foul play. Most likely it was an insider deal and a story to tell the public as to why they had to switch back to the original company for service, despite them being more expensive.
I mean, had the scheme worked right, the public rail operator company would have told the local authorities that they had to pay the losing bid because the winning local service provider couldn’t provide the service. To pull the trick off successfully, both companies would act in collusion and provide mutual kickbacks from the public funds.
Imagine if your Toyota Camry bricked itself if it detected you were getting an oil change from Firestone or Jiffy Lube instead of at the Toyota dealership. That’s basically what happened.
If you read the original story. The government contracted with a company that was using the train manufacturer’s service manuals to perform overhauls on the trains. They were completing service using the 15,000 page book from the manufacturer but when they went to turn start them they would fail. It turns out the trains had GPS modules and code to detect if they were in the vicinity of certain train depots (e.g. competitors) for too long and would brick themselves. After removing these modules and resetting the cleared parameters in the train computers, they magically started working.
This is pretty much already the case for cars. It’s not quite bricking the car but newer cars save an access log for the on-board computer/ECU and there have been cases where dealers will deny the warranty if it detects that any 3rd party has connected to the service port.
Which country? It would be extremely illegal in US.
No, even mechanical parts are susceptible to “pirate” parts or counterfeits. Maybe that replacement piece for your Deere seems fine until a certain temperature, or vibration mode the cloner wasn’t aware of when they chose the alloy for the counterfeit.
This business model of selling the hardware at a loss and trying to claw back the money by overcharging for “service” is flawed, and businesses that try to force the business model onto customers deserve to go bust.
Got to wonder where the suits and beancounters come from in companies these days… As they clearly have no concept that their customers might actually go somewhere else, or be aware enough that they object to be milked for their money. That short term bump for a quarter or two, maybe even a few years if your lucky and nobody notices quickly isn’t worth it in rather short order…
Apple can get away with it more easily 90% because they are a fashion brand and not that expensive, with a little hint of still being a good choice for audio work, convenient to use if you stay in their ecosystem and the like – the products might be overpriced, often low performance and locked down to prevent repair (though they are least making some of the right noises on repair now, even if its more hot air than substance) but while they are working they actually work well enough. Plus their product types typically have a relatively short expected life so anti-repair practices only shorten their useful lifespan in most users eyes a tiny bit anyway – it breaks, don’t repair it just go on to the new model with better performance, you’d have been doing so in the next year or two anyway…
But a train or tractor that you would expect to use for decades and need to maintain to make that lifespan possible?! Completely bonkers to think you could force a repair monopoly like that and not ruin your business in the end. Plus you would get many of the repair contracts to keep the trains you built running if you charged a reasonable amount and did a good job anyway, and most likely you can’t actually handle doing 100% of the repair work in house in a timely fashion to keep the customers happy… Putting in a polite request that appears once or twice every few years to say ‘you haven’t had a BRAND service in x years, we really want to be sure you are having a good experience with our products, so would you like to at x% off?’ type thing in your software or as a rep contacting the operator is about the most you’d get away with without overly annoying the customer.
So true. In the 90s I was involved in sales and service of an audio editing system that ran in Apple computers. The editor company would get the product dialed in and reliable on a specific model of Mac, then 9 months later Apple would end that model and there was a new model to wrestle with all over again.
That’s an issue with embedding your code within the hardware of a system instead of being able to use any system and riding your code on top of the OS. I realize that the tradeoff of running code on top of an OS layer is somewhat diminished capability, but if the reliability of well designed code can run across a number of newer systems and OS releases it’s worth the trade-off.
I’ve got 2 different vendors. One that’s on top of OS, that system runs like a top and any issues are easily addressed when they do occur (rarely). The other vendor has their code hooks deep into the pie and individual machines require changes to registry entries and other locations so deep that it takes days to sometimes sort out the simplest of issues.
If you can’t write code to be dynamic, consider another career.
>As they clearly have no concept that their customers might actually go somewhere else
With these sort of large infrastructure and transport projects, the answer is that the customer is the government and the whole deal is corrupted by cronyism.
In this case too, the rail operator is not a private enterprise, but a company that is owned by the local government. They have no choice but to buy what the bureaucrats and politicians tell them to buy. Even when there’s some sort of nominal bidding process, the procurement rules are fixed so that the same company always wins, or that company always makes an offer that nobody else can win and then goes “oops we went over budget and fell behind schedule” time after time, and they never get punished for it.
Most likely, this “dead man switch” was designed as an excuse to why the publicly owned company should buy repair services from the same company – and it was already decided by both parties how the deal should go – but someone along the chain of command went off-script and exposed the scheme.
But that is obviously not the case here. The very reason that Newag put the softlock in the firmware was that they lost a bid to a local train repair company, fair and square (Newag was more expensive). And the sale contract specified that 3rd party repairs must be allowed (so the manufacturer had to provide all specifications).
Exactly my point. They put a booby trap in the software as an excuse so the government would reject the third party and fold back with “Well these guys couldn’t do it, therefore we must pay NEWAG instead.”
But some employee who was probably not part of the insider deal went off the script and hired the hackers.
The “third party,” SPS, initially hired the hackers. I don’t see how you would insert them into the “deal” supposed to lock them out.
I didn’t. What I meant is that the rail operator and the train manufacturer were in collusion, only pretending to have a bidding contest for show, knowing they would have to switch to the train manufacturer once the third party (SPS) fails to fix the trains.
Which actually did happen – the original company was hired again to bring the broken train back to service – but then the hackers exposed the trick.
You do realise in many nations these bigger more expensive projects are bought in from Japan, Korea, Germany etc? Local corruption can absolutely exist and push business to a local firm when it exists. But lots of nations have not got the industrial capacity or more likely the demand to set up production to build their own trains – at least not in a cost competitive way compared to ordering in from the nation that is already able to just produce their trains to order. So with a company pulling this stunt in their own backyard even if its all ‘sanctioned’ by the local officialdom it probably will hurt their business and prove to be a bad move – nobody is going to buy Deere or Newag if they have other options while these BS moves are being pulled. And while you might have purchased your local Politicians buying the entire globe’s worth to force your BS on every customer you could have sold to…
There is a reason why almost all of Europe has Leopard II’s as a very currently relevent example – can’t afford the R&D, tooling or to keep production running enough to maintain a fleet of their own domestic design, so buy German as it at least comes close to matching their needs at a price they can more easily afford. And the more of your friendly neighbours that buy the same the more confidence you have in spare parts availability, the more certain you are the production line will still be there if you need to order more etc.
I wonder if the Leopard 11 has a ‘right to repair’ by 3rd party?
Not without Rheinmetall blessing https://kyivindependent.com/germany-says-it-reached-agreement-with-poland-on-leopard-2-repair/ hence why Poland bought >1K Korean tanks instead.
> it probably will hurt their business and prove to be a bad move
That’s assuming they get caught. If nobody notices the BS, they will come up smelling like roses and the OTHER companies will look worse.
No criminal would commit crimes if they thought they couldn’t get away with it.
With something like this you are 100% certain to get caught, the only question is if the suits that made those decisions have made their fortune and decided to move on to start their next company wrecking scheme somewhere else before the story breaks.
And then if they can in any way be held accountable afterwards or if the company is left taking all the pain on their own – strip those suits that came up with, and signed off on the VW Diesel emmisions test scandal or this of everything they own and maybe you put enough fear of loss into the greedy shits they will not make all their plays with the expectation they are going to move on in a few years smelling of roses after all the ‘bumper profits and massive bonuses’ leaving a ruin behind them that isn’t their problem.
Seriously? Apple’s repairs are quite reasonably priced, and the replacement batteries I’ve had from them last years, whereas the 3rd party replacements die within a year.
Their stuff lasts, and the performance is high. We retired one Mac at 10 years old, still a main workstation. I give them an expected life of 5 years (which they usually exceed), vs 2-3 for HP or Dell.
The world of Apple and genuinely high performance is rarely actually compatible – though it might function well enough. But the generic PC stuff and Linux (or even debloated Windows) of the same age is usually massively cheaper, more powerful and often has been rather more performance per Watt so cheaper to run too.
Their new silicon was really quite impressive and is really good at some tasks, but that doesn’t negate the massive overcharging for rather modest performance they have tended towards for a prolonged period beforehand. Nor does it really fix their problem of cost to performance. It is just the first time in a while they actually look competitive enough you might choose them for more than the inertia of staying locked in the Apple ecosystem.
And in this case ‘lasts’ is rather relative – you say retired at 10 years great you found a good one or a user that didn’t any performance! But also that you expect them to be replaced in 5 – so it doesn’t matter nearly as much if they have poor repair practices (And poor repair practices and prices they most definitely have) when the product was usually going to be at least half way through its expected lifespan BEFORE it needed anything. A very different case to the tractor…
I work for an ITAD & we refurbish & sell used office IT equipment. The majority are Dell OptiPlexes/Latitudes/Inspirons – from 1st Gen i-series onwards – that have had fairly hard-working lives. We get lots of HPs, some Lenovos, Stones & Viglens too.
99% go back into circulation & we give a 6-month warranty.
I call bull on your “2-3 years for HP/Dell”.
DRIVERS included has a knockoff multiplication effect on other replacement drivers thereby causing a lack of drivers.
So when is the CEO of the manufacturer going to get prosecuted for intentionally sabotaging public services?
As soon as investigation is finished…
That’ll be 6 weeks after he leaves with his golden parachute and retires to somewhere sunny then.
IF (and it’s a big “if”) the hackers are sued, I would expect the people at SLS to step up and fund their defense. I’d expect other users of Newag trains to step up as well, because if Newag gets away with this, the problem will only get worse.
I expect NEWAG to get a little slap on the wrist and a “don’t do it again”, and the local rail authorities keep buying their services despite the public outcry.
I’ve seen it happen so many times. There’s a certain IT company where I live, that seems to be the government’s go-to service provider when it comes to upgrading any IT systems from hospital data management to public databases and information systems. Every single one of their projects has been a failure, requiring extensive re-work and running many times over budget and being generally worse than what it replaced, yet they get hired again and again, year after year. Sometimes the name of the company changes, sometimes it merges with another company to hide their trails, but it’s always the same people. They always under-bid, over promise, then fail, and somehow the government can’t stop buying their services.
Manufacturers want a repair monopoly because they make good money off the repairs. Yes of course the official reason is “safety”.
Does HaD remember Taylor fight over their McFlurry machines and Kytch?
I’m told car dealerships have very little margin on a car’s sale, it’s the service dept. that brings in the real money.
Greed is good, greed is legal.
In many cases if you follow the path that leads up to a manufacterer’s DRM policy, at the source you will find a pack of greedy blood-thirsty Trial Lawyers.
“Safety issues!” cries a vile bunch of rats
If you dig into details this whole sage is quite terrible.
Newag agressively denies all and is making fake claims against the hackers, as well as shill social media posts defending them and attacking the hackers.
“Our software is clean. We have not introduced, we do not introduce and we will not introduce into the software of our trains any solutions that lead to intentional failures. This is slander from our competition, which is conducting an illegal black PR campaign against us – in the statement sent to our editorial office, NEWAG firmly denies the manipulated information of Onet and its interlocutors, i.e. representatives of a group of hackers hired by the competitive company SPS Mieczkowski. Newag adds that the servicing of previously delivered sets generates only about 5 percent…”
“It is not true that we caused faults in our trains to allegedly take over orders for their repair. This is slander. The company servicing rolling stock for the Lower Silesian Railways was unable to fulfill the order to service our trains and, in order to avoid contractual penalties, created this conspiracy theory for the media…
“… This is not the first time that we have notified law enforcement authorities that our software has been modified without our authorization.”
Cell modem UDP to CANBUS gateway, and not possible to do a remote F/W upgrade on the trains. Dragon Sector has proof you cannot simply change the code without the ability to re-compile. A train had F/W compared before and after a Newag repair- and it was changed (from 10 day to 21 day brick-if-parked delay). There’s a lot more but the object code speaks volumes and it cannot simply be “hacked” as Newag claims.
People are wanting to know about the train maker’s (GIT?) repo – what’s in it, should be clean if Newag is truthful. Otherwise there would be a legacy of code mods/changes to support the tyranny and fraud.
The exec’s and politicians involved need to be thrown under the train.
{I ran the rynek–kolejowy-pl story through google translate.}
“… threatened with legal action by the manufacturer, who are citing possible safety issues.”
Yes the safety of their wallets…
Not exactly tech-related but the news today said that police entered Newag’s premises and seized “data carriers for investogation purposes”.
We’ll see what the developments will.be