For most Hackaday readers the process of buying groceries this weekend has been a relatively painless one, however we’re guessing some of our German friends will have found their cards unexpectedly declined. The reason? A popular model of payment card terminal, the Verifone H5000, has suffered what has been described as a “software malfunction”. So exactly what has happened? The answer is as simple as it is unfortunate: a security certificate for German transaction processing stored on the device has expired.
The full story exposes the flaws in assuming that a payment terminal is an appliance rather than a computer and its associated software that needs updating like any other. The H5000 is an old terminal that ceased production back in the last decade and has reached end-of-life, however it has remained in use and perhaps more seriously, remained in the supply chain to merchants buying a terminal. With updates requiring a site visit rather than an over-the-air upgrade, it’s likely that the effects of this mess could last a while.
In case the hardware for this type of equipment interests you, we’ve had a teardown on another Verifone terminal in the past.
Verdammt!
Achte auf deinen Tonfall, junger Mann!
…Ich lieb’ dich.
https://en.wikipedia.org/wiki/Verdammt,_ich_lieb%27_dich
Okay, lass’ ich durchgehen ;-P
Top notch example for german mentality! For so many people in germany, technology is not a tool, it is more like some arcane magic.
Your experience of the world is entirely different from mine, and I’m not sure what a failure of POS terminals has to do with how the majority of people think about anything…
And Verifone is an American company.
If you have ever been to Germany, you will know that cash rules. Cards are accepted at very few places. People still visit bank branches to withdraw money via human cashiers. There is still no widespread acceptance for ATMs.
Wilkommen nach Deutschland.
Well done Germany, far more resilient to this than all those teendy fools in the rest of the world who want to swipe for everything.
a) I don’t know what Kuhkaff you visited (or when) but this is simply not true
b) It’s “WiLLkommen IN Deutschland” :)
I am sorry, but he is mostly right. Cards are accepted in a lot of places (but not as prolific as in most Western European countries), but the Germans just prefer to use cash.
My own experience: drive into a parking garage, no card terminal. There is a ticket machine that only takes coins. Walk to a store to change a tenner into two euro coins. It’s like going back in time forty years.
Once I had to pay an hamburger with a card because I somehow managed to lost my money, the cashier looked me really bad (in Munchen).
Elliot lives in germany. I think he knows :-)
Also that’s kind-of a good thing. We somehow have to keep cash alive (sadly).
My guess: you’ve been here before the pandemic. Since then I’m doing fine walking around only with my drivers license and my phone. No problem(*) paying with apple pay.
But before then – yeah. Quite a few places that did not take my card.
(*) I’m still keeping about 50 € in my phone’s case since, you know, to feel better ;)
I don’t see where he got it from either, but they have pockets of yokels like any country, it’s like basing your opinion of US on Kentucky or somewhere though.
A lot of the dystopian future movies show a mostly empty earth with the video displays still running canned messages and automated digital ‘jock in the box’ radio stations continuing to play top 40 music with time checks and automated weather reports. The pioneering heroes will be able to cross patch and use the satellites, radio stations and internet to restore humanity. Not so. Everything will be permanent blank screens and dead air, not because of the equipment, but all of the Smartnet and satellite modem configuration option files will expire because they haven’t had their annual or monthly license fees paid. I would imagine that after the big disruption (Godzilla, asteroid, little green men, global warming, etc.) the whole internet and satellite constellations will all go dark probably within 6 months.
The only thing that will still work is amateur radio.
Are you thinking about a specific movie or series of movies here? Perhaps I’ve just not watched enough movies, but I can’t think of any that match your description.
This. If you prepare a “rebuild civilization” box for your garage let it have an OpenWrt router and a bunch of USB sticks filled to the brim with technical ebooks. Maybe even with some backups. USB sticks and SD cards are dirt cheap. So we better stock up now, fill them with stuff and put them in a box.
Maybe we are one of the few “lucky” survivors and if not someone else stumbling over that box might make some use of the data within.
Your USB sticks have a brim?
with all the displays showing BSODs.
Standard problem – someone is copying openssl command from google. Example commands have almost always validity of 365 days. “Its too short!” – you think, changing to 3650 (almost forever – you think) and after ten years BOOM – you loose your openvpn network with hundreds of servers all over the country :)
True story, happened to me.
Someone we will not name here debugged a software for the better half of a day because it suddenly crashed somehow.
Then they realized, that this is the build in lifetime limit (to force updating) implemented by them. And commented with big bold banners.
They could weakly explain this by that the software heavily uses multi threaded design and the check is in a very late thread easily forgotten of. Or by medical reasons of to low caffeine levels.
But they didn’t. Celebrate their moment of stupidity.
Oldschool programmers (like me) who started out way before the internet or cryptography was of any relevance for most of us, we still sometimes tend to attempt writing software that can last forever as part of a maintenance free “black box”. Unfortunately, this doesn’t really work out as intended any more, as nowadays every bit of software has to connect to some sort of API (which will deprecate and go out of service at some point) using some sort of crypto/validation where certificates or keys will expire at some point. The days of not having to maintain certain parts of our systems are over. It frequently annoys me, but it’s the reality we have to deal with now. Software expires unless we actively work to keep it fresh.
Heh, need a version of this done for embedded systems…
https://pin.it/3eWblnD
like back to 8051 or something.
Y2K was a great example of how ‘We’ initially did not think about the long term value of what we were creating, and if it had an actual ‘Use By’ Date.
Y2K crept up on us, and early on some people started to realize the ‘Drop Dead’ cliff we were getting ready to fall off of..
The problem with Certificates, it they don’t all expire at once, they will sneak up and reveal themselves.
This just shows the old saying.. ” Don’t put all your Eggs in one Basket’ ( No matter how smart your basket is )
Cap
“And yet, they wanna make the certificate lifespan EVEN SHORTER!” – Me, on discovering upon renewal that I have to renew the external certificates every 10 months for stuff we’ve had for 10+ YEARS, and after having to explain to Upper Management that the certificate we paid that had a three year lifespan will now have to be re-purchased every year, and on different months and getting heavily grilled over the reasons and rationale of why it’s supposedly a good thing.
I know, I know, it’s all in the name of security, but the bulk of the certificate using systems I have (and the providers we get them from) haven’t heard of automation outside of their own internal systems, and require manual intervention every time something like this happens. (Hell, one of them requires me to update each server in the cluster with the same certificate manually; thankfully, the UI for it lets me do them all in one go, but still….)
You need new stuff or different providers
To err is human, to really screw things up you need a computer…
This is a shining example of the fragility of the epayment infrastructure.
A few hundred bytes can bring an entire economy to it’s knees.
Hopefully it will encourage people to reflect on the advantages of cash…..
… cash that you get from a cash machine that uses multiple security certificates and is connected to your bank through the internet.
Back to the matteres bank it is!
At least banks are maintaining their ATMs.
The funny thing is, banks actually earn money with each transaction made through payment terminals, and they bill maintenance services to the shops using them. Yet the only maintenance they offer is usually replacing the terminal if it somehow stops functioning, so it’s nothing like regular software updates and stuff.
I hope the shops claim compensation damages to the banks they’ve been paying for their maintenance.
But cash you can get in advance and have some at home for the necessity when the card infrastructure fails.
Recently went to a drive through and found that they couldn’t give any change for cash or take plastic at all. Why? Register/payment system had crashed. They could only accept _exact_ change. I was rather surprised they could accept even that.
As much as I like cash, it’s not a panacea for these issues. Businesses need to have backup plans that are fully and totally offline, preferably fully mechanical and dead-tree based. And the employees need to be taught how to use such systems. Missing either of these pieces ultimately means a full system failure. The internet _will_ go offline. The certificate _will_ expire. The device _will_ become obsolete without warning. The cloud service/api _will_ be discontinued. The edevice _will_ crash. What’s plan B to keep the business operating?
I am old enough to remember Y2K issues. Gas stations rejected cards whose expiration date was mistakenly assumed to be 1900s rather than 2000s, airport schedule off by 100 years, etc. Overall impact was minimal fortunately as most technology was fixed to correctly handle obsolete 2 digits year format or replaced with newer software/hardware to avoid this.
I also remember Y2K. Predictions abounded of planes falling from the sky like stones at the stroke of midnight.
Many systems which used 2-digit dates assumed that 90 was 1890, because when you’re building a system in the 1970s, that’s a realistic DoB. So the issue had been going on for a long time, with rolling epochs.
A related issue was systems that used 999 as an end of file marker.
There were systems impacted by Y2K, but they were relatively scarce, and only built by truly incompetent people.
Some of the “fixes” made the year roll over to 19100. IIRC there was one library system that issued a fine for a book overdue by 19,000 years.
You could write a movie about that, a Terminator style robot dispatched 19000 years ago to find a person and make them pay!… their library fees. Maybe it upgrades itself over the years, interstellar travel becomes a thing, it gets involved in politics or founds a social network to gather information to find the debtor.
I only directly encountered one thing that wasn’t Y2K ready and wasn’t fixable. A concrete plant which had an old 80286 for their accounting, using a DOS book keeping program.
Fortunately their computer guy was smart and setup a test file then changed the date and time to just before midnight, Dec. 31, 99 then let it run. In his words “It freaked.”.
I sold them a brand new desktop with Windows 98 and a copy of Quickbooks and suggested they start using it right away, don’t put anything else into the old computer, just hang onto it for tax and archive purposes.
My UNIX V6 system is full of files with invalid date codes, and if I do an `ls -l` it shows up as day numbers being random characters.
They didn’t have Y2K exactly, but the old 16-bit epoch rolled over in December 99, but it wasn’t that bad, it just jumped the date backwards.
Of course… I’m not running date critical applications on it….
I keep meaning to install the 32-bit time patch on it, but haven’t got round to it yet…
The advantages of cash:
Cash does not (yet) know:
– Who I am
– What I buy,
– Where I buy it
– When I buy it
– How often I buy it
(scenario: “.. You buy too much cola… your medical insurance is now more expensive..”)
Cash cannot be suddenly switched off or disappear if I become an “undesirable” person
– Check out the situation in China: https://nhglobalpartners.com/china-social-credit-system-explained/
Cash does not generate revenue for the “transaction parasites”
– This amounts to an extra tax that previously never existed
Cash does not inform every man & his dog of my bank account information
A cash transaction consumes no (or very little) electricity
– Sustainability??? Ah what the heck, I’ll just wave my card/smartphone for this chewing gum
IMHO epayment is THE biggest threat to personal privacy and personal freedom that has ever existed.
Privacy & anonymity is like virginity. Once gone, it’s gone forever.
To say “..this does not matter because I have nothing to hide..”
is like saying “… I do not believe in freedom of speech because I have nothing to say…”
I will use cash until I shake off this mortal coil
Cash subject to inflation and deflation worth only good as toilet paper. e.g. rubles. Cash worth nothing when your regime ends. e.g. confederate money. Cash not on the gold standard.
.. or when inflation has gone so far out of control that it’s literally cheaper to burn your paper money than to buy firewood. (Post WW1 Germany, IIRC)
The US Dollar hasn’t been tied to the Gold standard since the 1970’s, IIRC.
Atleast in the event of hyperinflation I can melt coins to sell for scrap metal and pulp paper to sell for paper making. I can’t do anything useful with 1s and 0s in a government or corporate run digital account.
You are missing my point bro…
ALL monetary systems are vulnerable to inflation.
Your bank account is eroded by inflation, regardless of whether you pay by cash or epayment.
My contribution relates to the privacy & anonymity dangers presented by epayment..
Cash indeed has its troubles, but all digital money is subject to the same troubles too/ Until you can offer me commodity backed cash, fiat cash is still better than fiat 1s and 0s in a distant server.
In the back of my head is the “conspiracy”theory that someone somewhere set those terminals up to fail intentionally to create the media coverage, awareness and so on.
Because for the last few years there have been a few political advances to get rid of cash (be it upper limits or what not) and this might increase the numbers of voters who are against that….
Nice conspiracy, but i’ll stick to my mantra:
„Unterstelle nie Bösartigkeit, wo Dummheit als Erklärung ausreicht.“
“Never Attribute to Malice That Which is Adequately Explained by Stupidity.”
I refuse to use any cards in parking machines for instance, because those networks seem very frequently compromised by black hats. The mentality seems to be “oh we’re only taking small payments, so we don’t have to be very secure”… Whereas *I* don’t wanna dip my monetary wick in anything that’s not figuring on protecting me up to my credit limit.
“They” seem to forget to add up as well, if you’re charging only a buck at a time for something, they figure, hey, what’s a lost or scammed buck here and there when we’re doing it MILLIONS of times a month… yeahnah that means it’s worth millions to hack you dumbass, and some criminal organisations will spend like 80% of that sum to enable a profit, this is a business to them now.
Indeed, I keep thinking with how required digital and card payments are becoming I should set up a few pre-paid or very limited cards for those places I’d rather use cash, but that don’t do so anymore… Sure you can compromise that payment method but the main debit/credit (credit card by default as here at least you have more legal protection to get reimbursed) card with its bigger pool is still ‘secure’. Also muddles up the privacy invasion bits somewhat – all the money might come from the same place and be easily traceable to SOME institutions, but its that extra layer of separation for those places not quite so free in data access…
Never quite been bothered enough, though now its getting harder to get cash from my bank account around here as well (ALL the cash machine and bank branches that there used to be 20 odd inside 15 mins walk are now all 40+mins away)…
“Cash cannot be suddenly switched off or disappear if I become an “undesirable” person”
Really, because I have some Confederate States money and German Reichsmark that say otherwise..
But they had to “switch it off” for all users, Trudeau was able to turn off digital money for only his political opponents. Hard currency is still more resilient, when a government makes hard currency worthless they hurt themselves too, they can make specific pieces of digital money worthless so as to only hurt opposition supporters.
100%
Programmed obsolescence and SSL certificates. More eWaste in the same of “security”.
100% this. Just like old ‘droid phones who can’t browse the internet just because there is no mechanism integrated to update SSL root certificates on the damn devices. This is 100% planned obsolescence in the pretense of “security” and 100% bullshit.
Damn right it is. That’s the real reason behind the switch to all this WEP stuff and SSL and stuff. ROT13 worked fine until manufacturers realised they could convince people to upgrade because of some phony “decryption” thing.
What speaks against just updating the certs and SSL/TLS libraries?
Can’t tell me that that old ARM in that old ‘droid can’t handle it.
I have a perfectly working WinCE PDA with Wifi that can’t access the web because of this. Yes, it is old but still good enough for many tasks. So why do we throw away so much still working stuff that might only need an update? Because updates make no money, you are forced to consume. Buy new, throw away, rinse and repeat. And the garbage pile grows and grows…
There is likely no technical reason these SSL certificates couldn’t be updated.
I used to use a Nokia N900 linux phone and if it hadn’t suffered terminal hardware failure I would probably still be pushing out unofficial updates to the core root certificate package (its used by the web browser and a bunch of other stuff). Although the fact that the web browser is using an ancient fork of the Mozilla codebase that’s too old to support the latest TLS standards and can’t access large chunks of the web probably renders any certificate updates mostly useless anyway…
Said tongue in cheek, but cryptography is a moving target with quantum sooner or later coming onto the scene. Bad guys don’t stand still, why should the good guys?
I am one of “your German friends” [1]. I feel particularly smug for having stuck to cash all that time.
On the certificate thing, I think a hard expiry date is really a bad idea (because it doesn’t model reality very well: a cert’s security just degrades bit by tiny bit). There should be a way to make increasingly annoying noises until it stops working for good.
In the case of the card terminals, they could have reduced the max transaction amount bit by bit until it reaches zero over, say, six to twelve months. This would have given people time to react.
This shows again that software and systems design takes a lot of empathy and quite some time walking in user’s shoes — something our industry is particularly bad at. No wonder our users hate us.
I don’t envy the folks working at stores.
[1] Strictly speaking I’m not German, but I am living here for quite a while
+1 for gradual degradation. It’s both a more realistic model of security, and a much better way to spot where problems will occur.
make that +2 for gradual degradation.
+3 on the degredation- We have internal apps that use self signed (SHOCK! HORROR!) certificates for the web GUI- outside from the browser complaining about it, it works just fine. (in fact, one vendor application we use breaks horribly if you try to replace the certificates it generates with something from, say, an internally trusted CA or an external certificate.)
I keep a wad of emergency cash well hidden at home just in case my cards stop working, internet stops, bank crashes etc.
However even that is about to become obsolete as the government has changed the design of notes and my old notes will soon not be legal tender. So cash too needs upgrading from time to time!
In the USA, all currency ever printed or minted under the authority of the United States Mint is still legal tender at its face value. So an old “Trime” can still buy three cents worth of something. Got an old Morgan Dollar? It’ll buy a large Coke at McDonalds.
Not anymore pal…
Currently the subject of intense legal discussion
see: https://safer-america.com/can-a-united-states-business-refuse-to-accept-cash/
Fun stuff. I’d always wondered what the whole “legal tender” thing was about. Short answer: only about debts already owed and how they can be demonstrably paid off.
“There is no obligation on the creditor to accept the tendered payment, but the act of tendering the payment in legal tender discharges the debt.” https://en.wikipedia.org/wiki/Legal_tender
And that has nothing to do with accepting dollars/whatever as payment. A firm can specify how it wants payment as part of the transaction beforehand. If you only want to trade goats for automobiles, nobody can force you to do otherwise. (Come to my shop with a herd, or go home.)
But if I already owe you $10, you are required to take those green things as payment.
In the UK you always get 6 months or more of warning when an old style note gets obsoleted, and banks still have to let you pay them in to savings accounts eternally after the date when they cease being tender for shops.
Hmm. that would be news to me. I could still change my few remaining Deutsche Mark into current Euros. Admittedly, I’d have to go to a bank to do so, but it remains a valid currency, even more than 20 years after the switch to euros. I would say that is enough time to change your mattress ;)
That’s why bitcoin wins.
Because when a bitcoin certificate expires it still works?
Yes, any paper wallets you had printed prior to Jan 1st 2020 or physical bitcoin are expired now, I’ll pay 10 cents a piece for historical value ;-)
Maybee it not the certificate…
https://www.golem.de/news/verifone-h5000-was-hinter-den-ausfaellen-der-ec-kartenterminals-steckt-2205-165742.html
Golem article is good.
Their summary: the device supports and end-of-life banking protocol (PCI-3) that was supposed to be phased out by now, but was granted an extension. There was a firmware update pushed out in December that updated this end-of-life condition, but many devices didn’t get it.
Verifone is saying that it’s a simple software bug. (“Don’t panic! Not security relevant!” they say, entirely unconvincingly, given that the error code reads “Cert not found”.)
The Golem article says that they’ve tested devices with and without the December update, and only those with the update work right now. And those without it cannot update because they don’t trust the update server without the cert, so that’s basically the status, whatever the root cause.
The German finance commission is looking into this, so my guess is that it will all become clear in about six months or six years, depending.
This should be a lesson for every manager: buy stuff that has long-term support if you plan on using it long-term.
Open source.
… would sign that. But then the company “xyz” should also keep a programmer at hand that, at best of times, helps keeping the code up to date. Not only copying and using it.
Works really well until the source is bankrupt or bought out
Then you can just fork it and keep going, right?
Might be talking about the other kind of source. Like in a source of goods and services.
This is why cash (coins and notes) are still the best method for face to face transactions. Just a shame there isn’t a way to make cash work for remote transactions too. Keep cash alive, for every payment you make in a physically present shop do it with cash.
Expected this article a few days ago ;)
What a POS!
There are shops and bars where I live that don’t accept cash. Needless to say, I walk out of there and clearly explain to the staff why I’m leaving.
A common refrain is “electronic payment is more secure!” Ask yourself, how many attempted electronic scams have you seen this week? Spam from Nigerian princes, fake emails “from the bank,” bogus phone calls “from amazon/the post office/your bank…”
OK, now tell me how many times you have been robbed of cash. I am willing to bet that for the vast majority of people (and businesses), the former number is larger than the latter.