What hacker doesn’t love a puzzle? We have a doozy for you. According to KUOW — the NPR affiliate in Seattle — they have been getting an unusual complaint. Apparently, if you drive a Mazda made in 2016 and you tune to KUOW, your radio gets stuck on their frequency, 94.9 MHz, and you can’t change it.
According to a post from the radio station, it doesn’t just affect the FM radio. A listener named Smith reported:
“I tried rebooting it because I’ve done that in the past and nothing happened,” Smith said, “I realized I could hear NPR, but I can’t change the station, can’t use the navigation, can’t use the Bluetooth.”
The station also reports that Mazda dealers in the area are getting flooded with calls for the last three weeks about the issue. There’s a theory about 5G smartphone deployment but, honestly, we aren’t buying it. Since is Mazda, we figure it has to be specific to that brand. However, it seems like it isn’t happening across the country, so there has to be something specific to KUOW, we are guessing. Maybe something in the RDS stream? What’s your theory? Maybe a Hackaday commenter will help the station and Mazda solve the mystery.
We know a famous fictional Seattle radio station whose on-air talent used the tagline: “I’m listening.” Perhaps KUOW should adopt: “You’re listening,” if they continue to capture car stereos. We’ve seen odd interactions mess with technology before. Sometimes it is as simple as a key fob.
Ars Technica has an explanation direct from Mazda that raises some more questions: https://arstechnica.com/cars/2022/02/radio-station-snafu-in-seattle-bricks-some-mazda-infotainment-systems/
> The problem, according to Mazda, was that the radio station sent out image files in its HD radio stream that did not have extensions, and it seems that Mazda’s infotainment system of that generation needs an extension (and not a header) to tell what a file is. No extension, no idea, and the system gets corrupted.
I would think a software reflash would solve the problem, but it’s possible the bootloader is hosed too. Need more fault injection testing.
And this is why service ports are important. To have the ability to go in and operate the OS at the command level to correct for buggy software is CRITICAL. The radio station should fix the issue of no extensions however. In fact Ibiquity should fix the bug in their software that allows for files without extensions to be used for images. Maybe do a header check and put the appropriate extension on a file that lacks the information?
I’ve had to build a system to send icon files to the Analog and HD encoders for an FM station. After e-mails galore I finally got the secret from Ibiquity and the encoder manufacturer to get things to operate. Trust me the ibiquity software is a bloody nightmare to navigate..
Sounds like plenty of blame to go around! 🤣
Some of the comments suggest that certain character sequences in RDS streams can also prang up these infotainment systems. (“99%” is problematic.)
In Australia, we do not have “HD Radio”, so we theoretically could be safe from this vulnerability… HOWEVER, like much of Europe, we do have DAB+, and DAB+ can pull the same stunt (serving up files) as HD Radio.
In DAB+, there is a field that describes what kind of file is being sent (PNG image, JPEG image, MPEG video [yes, some countries use DAB+ as a television standard], Java…), including one type code (0, 0) which indicates a “generic object”; application/octet-stream.
Some stations (looking at you 4KQ and 97.3) send their slideshow slides as “generic” objects, and let the radio try and figure out what sort of file it’s looking at.
Question is, are devices detecting the file type by simple string pattern match, or are they looking at magic bytes in the file header? If the latter, they should theoretically be fine so long as the header is well-formed.
I think Mazda’s infotainment system though are just looking at the file extension and “trusting” it to be correct… a really silly thing to do given I can technically (but not legally) walk-up with a netbook and a HackRF One and “narrow-cast” any arbitrary FM/DAB+/HD Radio signal I care to emit.
I think this was a marketing stunt that went horribly, horribly wrong – as in “god as my witness, I thought turkeys could fly!”
In case you’re too young to get the reference:
Forgot the link: https://m.youtube.com/watch?v=lf3mgmEdfwg
So the person making the images for the radio station is using a Macintosh? Apple’s desire to be incompatible with the rest of the computing world strikes again? See the Apple Double file format and how huge of a PITA that was to use with other systems, along with HFS, HFS+, AppleShare IP, AppleTalk etc. Not using filename extensions was the least of the issues in doing cross platform work with a Macintosh. At least that was easy to partly take care of simply by typing in a . (on the Mac) followed by three letters, or more if dealing with newer PC OSes.
Your knowledge is a little out of date. Macs have used file extensions since the debut of Mac OS X in the year 2000.
True. Now Apple’s embarrassing offense is to encode images upside-down or rotated, instead of using the phone’s orientation sensor to put the pixels in the right order. Then they set a flag in metadata, and expect every piece of image-viewing software on the planet to read it and rotate the image EVERY TIME it’s displayed… instead of Apple simply writing it out correctly ONCE.
> expect every piece of image-viewing software on the planet to read it and rotate the image EVERY TIME it’s displayed
FWIW: They do it on purpose and not for a technical reason.
Microsoft does the same thing with the .bmp format and does it completely backwards from most pixel based formats and screen raster movements. They start at the bottom and move right to left, up the screen.
I was writing a bit of code in assembly language to convert graphics and was 1/2 through dealing with the .bmp nonsense, gave up, and just went with the .png format instead.
Not as simple as the station “accidentally” having their traffic/news/weather switch stuck on?
That used to be a a common “accident”, of the “oh, sorry, we labelled the switch the wrong way round, won’t do it again” type
Gives a whole new meaning to “Captive audience” .
That wouldn’t/shouldn’t crash the entire car “entertainment center”.
*sulks off mumbling about casette decks, 2 radio stations, back in my day, new fangled thingamajigs, good old FM bahumbug*
Quit you’re mumbling kid. In my day we had to listen to the AM radio! 🤣📻
Huh, this reminds me a LOT of the Roman Mars Mazda Virus from a few years ago (https://gimletmedia.com/shows/reply-all/brh8jm), where listening to the 99% Invisible podcast would crash the entertainment system, most likely because of the percent symbol in the name. Sounds like Mazda might want to write some unittests for their firmware!
That’s exactly what I thought of too. If it’s really an issue with invalid input crashing it, that’s a big bummer and possibly recall material, depending on whether or not the backup camera is also impaired.
Testing? We don’ need no stinkin’ testin’! (Adapting a Mel Brooks script)
On a serious note, if this is the quality of their firmware, I wonder what other wonderful discoveries remain to be made in Mazda vehicles.
> I wonder what other wonderful discoveries remain to be made in Mazda vehicles.
A transmission cooler line for a CX-9 that should be 3/8″ (11/32 to 3/8) at both ends and $2-$3 a foot (less then 12″ long). But, the hose is made with a 9 MM ID at one end and a 10 MM diameter at the other end, forcing you to buy the Mazda OEM part. Though if you are unlucky enough to have a CX-5 instead of CX-9, then no soup for you. Because the part is discontinued!
https://www.jimellismazdaparts.com/showAssembly.aspx?ukey_assembly=301239&ukey_product=1913080#61215B
19934 Hose, Oil. 061101 -. Required: 001. (Current price) $ 22.57
x7 mark up.
Could you use 9mm hose and then enlarge one end? I’m thinking maybe a brand new razor-sharp drill bit? Or, heat the shaft end of an old worn out 10mm bit some (definitely not red hot) and let it melt out a new, larger hole? Just thinking out loud, so to speak.
Hi,
I have drilled out low pressure gas lines for my own trucks before so I would not have to buy special fuel line. BUT, this was someone else’s car and I could not afford to hack it. Failure is not an option when the replacement transmission is north of $5,000.
The heated drill thing would not work on many hoses because (especially EFI) they have a special liner for pressure containment and to protect the rubber from the chemicals.
This started out with “I only like to work on Ford trucks before 2005, as I tell everyone”. ” Take it a dealer or something to get the oil change.”
She took it to some service shop for an oil change and they told her it had a transmission leak and they could “diagnose” it for $200.00 to find the leak. I drove it up on ramps and immediately saw the rubber hose was leaking and swelled from being old. Told her what it was, she then took it to a Mazda dealer and they wanted $400+ to change it and even more to do a transmission fluid change at the same time. I have not worked on a Mazda in 25+ years. It took me about an hour to get the car on ramps (again) and jam a 3/8 EFI hose on for a quick fix.
So, I look up the hose no one can find at the Mazda dealer (they want to sell the whole $122 metal and rubber line?), using their own website, send her there to buy JUST the hose, they insist she has to pay $10 extra for $33, even though they have it listed for $23 something right on the dealer website.
They really do try to take advantage of (blonde) women in repair shops.
“CX-9 that should be 3/8″ (11/32 to 3/8)”
No, most definitely it should not be 3/8″. In fact nothing SHOULD be measured in either fractions or inches, or worse, both. I agree it is stupid to have it 2 different diameters though. That’s just a dick move.
Actually Mel Brooks got that from Treasure of the Sierra Madre (and he got the line wrong)
https://www.youtube.com/watch?v=4OcM23Hbs5U
Ha, you beat me to it!
What’s a traffic/news/weather switch?
Some vehicles will quietly scan all stations for if one has a ‘news’/’traffic’/’weather’ flag set on their radio data, and can be configured to auto-tune to that station for the duration of the flag. Typically used to be able to send a traffic alert out, for instance.
No, the firmware and the bootloader is fine. The image file has been saved in the cache, the firmware attempts to read it on boot, crashes, restarts, rinse and repeat.
So they could fix the underlying bug (getting hosed if there’s no extension) and flash that, instead of replacing the whole unit as the ars Technica article described, except that who knows where the source is for a 2016 car, or the engineers who know it… I’d love to see one of those post-accident style reports on this, with how the problem happened (missing test case) and what mitigation techniques were in place and failed, or were missing (clear cache on boot loop? Why caching before verifying readable?), Etc. But I’m sure even if such a report is made, it will be internal only.
My cheap TV sometimes locks up on one of the non-English stations. I speculate that there is something in the Close Caption data from that station crashed the firmware as neither the remote won’t work nor the buttons except On/Off would work.
https://github.com/Trevelopment/MZD-AIO/
The original firmware we can’t mess with, but there is a version you can fiddle with
It’s been terribly hard to get info on which system(s) are affected; is MZD the one with the HD radio bug?
Could have been a disaster of biblical proportions. It could have been a Polka station.
We are loosing the oldest station in Indiana WBAA to Indy Public radio. We have more Christian radio stations than commercial but no community station unlike the other University towns around the area. Volume wars on all formats. It really is a disaster! The Hammer does what you expect only 2 channels away from the NPR FM replacement of that 100 year old station WBAA AM which has suffered from Heavy Damage, Hearing Difficulty, High Distortion, and Hidden Diction since ’07.
Piotrsko said: “Conservative talk radio on the other hand……..”
Please move that comment over to Ars please.
The affected modules were likely designed by either Sanyo or Visteon’s low-cost design team, any automotive electronics in low and mid level cars designed after ~2008 are done by these low cost teams which are usually centered in the “cheapest engineering labor possible” locations. You get what you pay for. No worries, though, these low cost designs also use as many chinese parts as possible, so they will all fail soon either way.
The official explanation essentially says that they didn’t sanitize their input
see: https://xkcd.com/327/
I learned a long time ago – any digital comms that connects to radio WILL be exposed to all sorts of non-compliant bit streams. You have to make the radio software robust, or else it will eventually fail. This is one of the unlucky ones where the failure isn’t just an assert and reboot, it affects non-volatile storage.
It seems the failure is a case of assert and reboot, except that it also fails during boot. In a loop.
They probably just neee to reboot 100km away where the radio station signal isn’t there to corrupt everything again upon reboot.
My peugeot 2008 (first batch) also suffers from data partition decay, after some times you need to do a factory reset to avoid crash loop when reading usb sticks.
But at least eventually it does goes out of crash loop when removing the stick.
What hacker doesn’t love a puzzle?
Well, this problem started when KUOW dropped “Says You” from their programming.
Go figure.
This is a major drag for the typical Seattle resident who spends half their life stuck in traffic and the other half looking for a parking space.
Always validate user input.
This is my First Rule of Programming. In this case, the radio station is the user and the radio software needs to validate input. Depend on the extension? That is user-supplied input – a careful programmer will treat the extension as a hint and validate the contents. This is just a variant of Little Johnny; Drop Tables.
Second Rule – generate strongly compliant output but tolerate loosely compliant input.
In this case, anticipate odd characters, missing extensions, and bad headers. Ignore what you cannot understand (rather than crash/loop/fail). This was good advice in the early days of the IETF and the Internet, and it remains good advice today.
Third rule: assume the developer is incompetent and needing a condescending lecture, neglect the possibility of a compiler bug or other build failures.
Wondering if this presents another attack vector into the car. Wireless HD radio stream input, not too difficult to create. Not just a missing filename extension but fuzzing data streams…
“Testing? Isn’t that what users are for?” Seems like not only Tesla has this idea.
You must be new to cars, using customers to test has been standard practice for decades. In particular, GM used customers to debug the 8-6-4 engine, they sold the Citation with basically no testing done. Mazda had no clue about the reliability of wankel engines but they sold them anyway. The Corvair and the Pinto were experiments to determine acceptable customer death rates. Yes you are a guinea pig.
I bought a new 1981 Camaro with the V-6. It was a total piece of junk and was in the shop 1/2 the time I owned it, less then a year. When it lit itself on fire in the middle of a snow storm from a clogged converter, I left it at the dealer, and started suing GM, I went writing right to Roger Smith.
As part of the deal, they said pick any used car off the lot and swap. So … I picked a 1980 Citation X-11 with the 2.8L, yellow with black stripes. About a year later I was working at a Chevy dealer selling Chevy trucks. I am glad I was, because the Citation was a piece of junk too. It was the first car I ever had that had free life time replacement of brakes, because they were under Federal recall for being a defective design.
> the Pinto were experiments to determine acceptable customer death rates
I worked as a Ford mechanic and changed 100s of Firestone ATX tires. Why? Because Ford under inflated their Exploder tires so they would not flip over to lower the COG, to fix the inherent design flaw that existed since the Bronco II (I own one). The under inflation caused the tires to overheat on long trips and explode.
> Yes you are a guinea pig.
Agreed.
The discussion on https://news.ycombinator.com/item?id=30268000 concludes that non sanitized input in the parsi g of station logo files is the problem…
And even with links to SDR projects that can emulate or send HD information for further testing
Working on such further testing right now.. :) This is gonna be fun.
What’s obnoxious is that Ibiquity tried so hard to keep the spec closed, there are no third-party sources of test signals as far as I’m aware, so there’s surely tons of other crappy code to be found, because it’s simply never been tested with anything but happy-path reception.
I wonder if a cold boot would solve the problem. By cold boot I mean disconnect the car battery and wait a few minutes before reconnecting.
The big mistery to me is “What the fuck is an NPR?”.
Whew! i guess I lucked out, since I have a 2015 Mazda and no plans to visit Seattle.
https://theneedling.com/2021/12/30/kuow-says-the-subarus-are-next-if-if-you-dont-become-a-sustaining-member/
This reminds me of an episode of 99% Invisible where names with certain ascii characters would not play on Mazdas.
https://99percentinvisible.org/episode/the-roman-mars-mazda-virus/
When I was a kid, and well into my teens, EVERY attempt I made at analog audio circuits, even the simplest op-amp headphone driver straight out of Mimms, picked up KUOW loud and clear, no matter what I did to try to shield it, clean up the power supply, shorten potential “antennae” (long wires), etc. I ultimately gave up on analog circuitry for decades.
NPR, Every Time.
Now how the heck a simple circuit like that gonna decode FM?!