Certain manufacturers seriously dislike open-source firmware for their devices, and this particular hack deals with quite extreme anti-hobbyist measures. The Meraki MR33, made by Cisco, is a nice access point hardware-wise, and running OpenWrt on it is wonderful – if not for the Cisco’s malicious decision to permanently brick the CPU as soon as you enter Uboot through the serial port. This AP seems to be part of a “hardware as a service” offering, and the booby-trapped Uboot was rolled out by an OTA update some time after the OpenWrt port got published.
There’s an older Uboot version available out there, but you can’t quite roll back to it and up to a certain point, there was only a JTAG downgrade path noted on the wiki – with its full description consisting of a “FIXME: describe the process” tag. Our hacker, an anonymous user from the [SagaciousSuricata] blog, decided to go a different way — lifting, dumping and modifying the onboard flash in order to downgrade the bootloader, and guides us through the entire process. There’s quite a few notable things about this hack, like use of Nix package manager to get Python 2.7 on an OS which long abandoned it, and a tip about a workable lightweight TFTP server for such work, but the flash chip part caught our eye.
The flash chip is in TSOP48 package and uses a parallel interface, and an iMX6.LL devboard was used to read, modify and flash back the image — hotswapping the chip, much like we used to do with old parallel-interface BIOS chips. We especially liked the use of FFC cables and connectors for connecting the flash chip to the devboard in a way that allows hotswapping – now that we can see it, the TSOP 0.5 mm pitch and 0.5 mm FFC hardware are a match made in heaven. This hack, of course, will fit many TSOP48-equipped devices, and it’s nice to have a toolkit for it in case you don’t have a programmer handy.
In the end, the AP got a new lease of life, now governed by its owner as opposed to Cisco’s whims. This is a handy tutorial for anyone facing a parallel-flash-equipped device where the only way appears to be the hard way, and we’re glad to see hackers getting comfortable facing such challenges, whether it’s parallel flash, JTAG or power glitching. After all, it’s great when your devices can run an OS entirely under your control – it’s historically been that you get way more features that way, but it’s also that the manufacturer can’t pull the rug from under your feet like Amazon did with its Fire TV boxes.
We thank [WifiCable] for sharing this with us!
(Ed Note: Changed instances of “OpenWRT” to “OpenWrt”.)
yup, putting in a flexi connector instead of the 0.5 mm pins is clever.
Until I read the linked article, I thought they were using the FFC connectors as a TSOP socket – that _would have been a cool hack!
+1, i have to remember this. I wonder if there are already PCBs with flatflex-connector to TSSOP-48 for the other end of the cable readily available on the usual platforms.
I came here to see flashing boobies…
Left disappointed
B^)
I feel like Louis Rossman would like this, given that HaaS is inherently anti-R2R.
As always I hoped you learned something and have a great day. He’s a class act
It seems it would be time that the EU mandates that manufacturers of any halfway smart hardware make it very clear whether you’re buying:
– a device; in which case they should not be abke to make any alterations that negatively impact any features (advertised or not) but still be responsible for security updates, or
– a service; in which the manufacturer could change a devices functionality provided you had the option to stop buying or get your money back if the service was for a fixed term.
If the EU makes regulations like this, most manufacturers will sell such a device worldwide.
I believe it’s very clear with these devices. They’re enterprise devices, where the ability to enable extra ports or speed remotely and without swapping kit or even downtime not just trumps any concerns about HaaS costs, but is a selling point.
It’s almost like two different worlds. Not that right-to repair isn’t important for businesses. e.g. farmers, etc.
I’ve been on the high end enterprise side of this and yeah, companies like Cisco are massively about selling you everything as a service / add-on.
I’ve managed thousands of MR-33s over the years. You buy the access point, but you cannot use it without a license which is sold separately. Originally there was a single license sku but there were discussions about creating a basic and enterprise license similar to what they do on their MX appliances. They just enabled software features, no hardware changes.
These APs typically have a 5 year life cycle so the MR 33 is about to start hitting the scrap heap. Cisco won’t let you claim a used AP so they all get junked and Cisco makes more money. That’s why the interest in OoenWRT on these. I ran it on some MR-16 and MR-18s a few years ago and pushed configs with Ansible at home. It didn’t have all the enterprise features but worked pretty well.
They’ll sell different versions for different market, as they do with many consumer and business electronics for the North American market. Many smart TVs in Europe and Asia have DVR capability to save OTA broadcast to USB storage. In the US and Canada we don’t get that. It’s disabled in the firmware. Many models of cellphones sold here have less RAM, less storage, slower CPUs, and lower resolution displays – at least one or more of those will be worse here.
Back when it was a cool thing to have composite video out the headphone jack by using a special cable, North American versions of those phones had that disabled. Same for FM radio reception using a headphone cord for the antenna. We never got a smartphone with an analog TV tuner, as was very popular in Asia, and since the switch to digital Asia has that in phones but not North America. There are USB ATSC tuners for both Micro B and USB-C with OTG but that’s a clunky kludge vs having it built in.
It’s done with vehicles too. For the 2004-2009 Toyota Prius, other countries got height adjustable driver’s seats, but North America got stuck with seats fixed at a perfect height for short Asians. Canada did get an option for rear disc brakes, but the US wasn’t allowed to have that. We did get an HID headlamp option not available in Canada, but hardly anyone bought that.
So no matter what regulations the EU puts in place, North America is still going to get the crappier versions.
There needs to be something done to fix that. It isn’t right, especially when it is done just to limit the functionality of the US models.
Who knows. Products have to navigate a sea of regulations and interests.
If a manufacturer can legally brick your device to prevent modification, then “Right to Repair” is just a bad joke. Am I missing something, or is there a solid reason to allow this practice?
Yes – imagine with this John Deere fiasco if they started using code that detected when unauthorized access to the firmware was made and it bricked the combine. It’s why I’m so resistant to new technology. I will own what I own as much as I can. Similar to my rooted phone where I can download my e-books, save the file, convert to pdf, and now I OWN the file. The more technology advances, the more I go the opposite direction. By the time I’m ready to kick it, I’ll be riding a horse to work and telling people that God would have given man wings if he intended for him to fly.
I recommend epub or mobi over pdf
PDF is an Adobe product, after all.
At least it’s nice that John Deere can disable a tractor after it have been stolen in Ukray and with great effort been transported several countries away.
I’ve been flashing open firmware onto my devices for many years, and honestly it’s never as good buying a device that’s actually purpose made for developers and engineers. I have Raspberry Pis still getting updates nearly a decade after they were released, whereas every time I’ve flashed an unsupported firmware it only extended the lifetime of the device by a few years at most.
If a manufacturer is locking down the device there is usually good reason for that, and if nothing else it’s usually an indication that the manufacturer doesn’t want to be responsible for answering questions or fixing problems that arise from user firmware flashing.
But especially as it pertains to Cisco, do keep in mind that handling computer networking can be a matter of national security. Cisco contributes a huge amount of the hardware behind the Internet, and has a huge responsibility to keep details of the inner workings of their infrastructure away from prying eyes. And they’ll go as far as booby-trapping their hardware to do it, that’s just how important security is to them.
Failing to respect Cisco’s reasoning and dedication to security could lead you down a rabbit hole you will regret. It’s all fun and games until you crack open the wrong firmware and post your findings online, and are summarily raided by the FBI.
Which nation’s security, exactly? What a load of puffed up nonsense. There is no security through obscurity.
Wow, what a FUD. First security trough obscurity NEVER works, especially not against TLA (three-letter-agencies) from other countries. Also it’s a good thing the RaPi gets still upgrades, because (sadly) there are always new (security) bugs discovered that need to be fixed. Most of the closed stuff will get upgrades for a few year at most (just look at smartphones). Do you work for Cisco or the NSA btw?
Is there any security that works against TLAs?
Yeah… I would say we should try to make it as hard as possible for them. Encrypt stuff using open source software and public algorithms, keep everything up to date and so on. Of course “they” can always catch you and hit you until you tell them your password i am afraid. (insert xkcd here) :-(
If they bribe the right person at the chip factory, then all the Kings horses and all the Kings men are not going to put Humpty Dumpty back together again.
Most TLAs have one problem: Assume, you have a encrypted device with state-of-the-art security. Now someone indicts you and evidence is that suddenly decrypted device. Proof that state-of-the-art is unsecure now. This info will spread like a wildfire.
Standard problem of all secret agencies. Indictment might reveal their technological potential, source of information etc.
I’m pretty sure that’s when they say a random kid knew a trick and everyone thinks there smart for circumvent an absurd amount of security features and no one thinks much about it past that.
The hardest part about actually broaching the subject of secret agencies is getting people to even consider let alone accept that they exist in a malicious context because most people would rather not even think about something like that and thus avoid the subject all together irregardless of how legitimate or illegitimate it might be.
People in such situations tend to fall out of buildings or suffer heart attacks or they get mowed down while crossing the street, they don’t go to trial. Nobody ever finds out.
If they can hide inside your USB controller, there is no hope. Every chip in your computer has firmware and you can’t trust any of it. Ethernet cards, display cards, flash drives, etc. They all have direct access to everything, and they can do whatever they want to your computer, and there is no way for you to even detect what they are doing. These chips all have special high speed IO circuitry. If you hook up scope probes you will change the impedance and the malware will hide and you won’t see anything. Same in software, they detect debuggers and tracing tools and they hide.
The whole thing is massively broken if there is no way to have a computer you can trust. There is no way to fix any of this without a radical rethink of what computers are and how we make them and a frank realization about the nature of the humans involved. We can make bridges and buildings that we can trust, despite unscrupulous contractors and human frailty because we have a system that doesn’t trust contractors and doesn’t trust human nature.
MR33 is an enterprise grade AP. It’s not meant to be hacker friendly.
Target audience don’t want some visitor to flash their firmware on it easily, and have difficult to detect foothold in their network (with wireless external access!)
If you want OpenWRT AP just get something else, there are better options available.
Nice hack overall.
“If a manufacturer is locking down the device there is usually good reason for that, and if nothing else it’s usually an indication that the manufacturer doesn’t want to be responsible for answering questions or fixing problems that arise from user firmware flashing.”
Good reason like making you pay for new hardware (probably from a different manufacturer after getting burned) when they fail to offer updates for security holes found.
Does anyone flashing an alternative firmware seriously contact the manufacturer for help/support?
Flashing an alternative firmware comes with a chance of ending up with a brick. With this simply looking at the device wrong guarantees a brick.
Sorry Simple User, As it has already been pointed out, “security by obscurity” is not security by itself. I guarantee companies that really matter to national security probably arent using routers you can pick up at walmart.
“fixing problems that arise from user firmware flashing” This is why those little “warranty void if opened” stickers are there. As soon as someone calls into cisco and says “i loaded openwrt on my router and now it just has a flashing power light” the proper response is “sorry, you installed custom firmware, you can ask the firmware writers for help. we are not responsible for modifications that take the hardware outside of intended scope”
“And they’ll go as far as booby-trapping their hardware to do it, that’s just how important MONEY is to them.” Fixed that for ya.
“crack open the wrong firmware and post your findings online, and are summarily raided by the FBI.”
So which conspiracy theory do you support? “the government pays companies to hide back doors for them” or “The government has hackers who discover 0 days and hold onto them for aces up their sleeves”? Either way, the better option for everyones security is researcher discovers vulnerability, the vulnerability gets its CSV, and is patched before something bad can happen.
Those “warranty void if open” stickers are unenforceable. Many a lawsuit has been won over this FEDERAL law.
Read up on the 1975 Magnuson-Moss Warranty Act for more info.
How much does Cisco’s PR department pay these days?
Very impressive, nice work. But a person has to ask why you would even buy such a device in the first place. Vote with your dollars and refuse to buy stuff like this. But perhaps the hacker (like I might have) bought it and never anticipated such outrageous practices by the evil Cisco.
You make a very good point. All those IoT gadgets we have — the manufacturer can pervert them at any point they choose through a firmware update. Distributed DoS – done. Local exploits for other devices on the same LAN – done. Scarier still – exploits to mobile devices on the same LAN, potentially providing real time location tracking. Anyone who isn’t segregating their networks is vulnerable, roughly at a guess 99.9% of the developed world.
What type of segregation would you consider appropriate here? like separate subnets?
Originally Meraki gave the hardware away for free, as their software requires a paid subscription to their cloud MDM to use it.
They do sell the things too so I can’t say if that’s what happened or not in this case.
But I fully see the allure of repurposing free hardware after electing to not enroll it for the monthly fee.
Remember that Meraki pre-cisco was actually a very good company, and only started doing acting this way once Cisco bought them.
(I still have a grandfathered free-100-device MDM account from before the acquisition)
High time that the law was enforced against them. It is a clear violation of the crimes act. A few firmware engineers and their managers in the dock defneding criminal charges, and it would be a lot harder to get staff to do this shit.
Every one is liable to imprisonment for a term not exceeding 7 years who—
(a)
intentionally or recklessly, and without claim of right, destroys or damages any property in which that person has no interest; or
(b)
intentionally or recklessly, and without claim of right, destroys or damages any property with intent to obtain any benefit, or with intent to cause loss to any other person.
The hardware In question is provided as part of a service and not sold to the consumer, ownership rights and interest remain with the service provider. Cisco pays its lawyers quite well and they know exactly what the law says and exactly how to use it to their advantage.
I’m sure that governments in Iran, China, Russia, etc are very impressed by those lawyers in de US.
That response only flies if the customer doesn’t pay for the hardware. Even then, if Cisco goes with the “the customer is paying for leasing the hardware,” then it needs to be made clear and evident when initially acquiring the device. Basically, it comes down to informed consent.
I’m not worried since it will be the eventual environmental reforms that change this practice, not the device ownership laws.
Personally, I think Cisco should simply just lease the hardware for $1 plus the cost of 2-way shipping when the license is purchased, requiring it to be returned when then license term is concluded without a renewal. Then Cisco becomes responsible for replacing broken hardware (plus the shipping costs) for the duration of the license. Further, they can opt to ship the customer a new, refurbished, or even an updated unit at their own discretion.
There. Problem solved. Cisco retains legal ownership of the devices and we aren’t saddled with replacement or disposal of failed or retired devices.
I lucked into a pile of sonic wall security appliances but the previous owner had wiped the flash off of them, but one would not even come up. I lucked out, they used a plug in radio assembly and it had been knocked loose but not out and a few of the pins were shorted. I re seated it and that one had one of the last flashes on it and it even had the means of copying it off, so that got me up and going with the lot of them. Sadly sonic wall sucks in that they totally software limited the hardware. I had thought about trying to get around that as I suspect it would not be super hard, but I had better pieces fall into my hands so now the old blues are just things to be hamfested off.
There are a bunch of new, high performance ARM router boxes with gigabit Ethernet and real processing power, for super cheap. The latest openwrt supports some of them.
The correct spelling is OpenWrt. Capital O, capital W, the rest is lowercase.
….but why? OpenWRT looks way better.
BTW: There are kinda better or at least other ways to achieve the same:
https://github.com/riptidewave93/LEDE-MR33/issues/13#issuecomment-1100838849
Rooting is recycling – breaks my heart how much hardware goes to landfill / environmentally dubious “recycling” in far away lands because the manufacturer can’t be bothered to support it anymore but also won’t let anyone else have a go with it.
Hey Editor, the official name is “U-Boot” not “Uboot”