ChatGPT, Bing, And The Upcoming Security Apocalypse

Most security professionals will tell you that it’s a lot easier to attack code systems than it is to defend them, and that this is especially true for large systems. The white hat’s job is to secure each and every point of contact, while the black hat’s goal is to find just one that’s insecure.

Whether black hat or white hat, it also helps a lot to know how the system works and exactly what it’s doing. When you’ve got the source code, either because it’s open-source, or because you’re working inside the company that makes the software, you’ve got a huge advantage both in finding bugs and in fixing them. In the case of closed-source software, the white hats arguably have the offsetting advantage that they at least can see the source code, and peek inside the black box, while the attackers cannot.

Still, if you look at the number of security issues raised weekly, it’s clear that even in the case of closed-source software, where the defenders should have the largest advantage, that offense is a lot easier than defense.

So now put yourself in the shoes of the poor folks who are going to try to secure large language models like ChatGPT, the new Bing, or Google’s soon-to-be-released Bard. They don’t understand their machines. Of course they know how the work inside, in the sense of cross multiplying tensors and updating weights based on training sets and so on. But because the billions of internal parameters interact in incomprehensible ways, almost all researchers refer to large language models’ inner workings as a black box.

And they haven’t even begun to consider security yet. They’re still worried about how to construct obscure background prompts that prevent their machines from spewing hate speech or pornographic novels. But as soon as the machines start doing something more interesting than just providing you plain text, the black hats will take notice, and someone will have to figure out defense.

Indeed, this week, we saw the first real shot across the bow: a hack to make Bing direct users to arbitrary (bad) webpages. The Bing hack requires the user to already be on a compromised website, so it’s maybe not very threatening, but it points out a possible real security difference between Bing and ChatGPT: Bing gives you links to follow, and that makes it a juicy target.

We’re right on the edge of a new security landscape, because even the white hats are facing a black box in the AI. So far, what ChatGPT and Codex and other large language models are doing is trivially secure – putting out plain text – but Bing is taking the first dangerous steps into doing something more useful, both for users and black hats. Given the ease with which people have undone OpenAI’s attempts to keep ChatGPT in its comfort zone, my guess is that the white hats will have their hands full, and the black-box nature of the model deprives them of their best hope. Buckle your seatbelts.

Years Don’t Dim The Shine Of These Curious Gadgets

[Maarten Tromp] recently took the time to document some of the unusual and creative electronic projects he received as gifts over the years. These gadgets were created in the early 2000’s and still work flawlessly today. Two of our favorites are shown here: Hardware Tetris Unit (shown in the image above) and Heap of Electronic Parts.

The “Heap of Electronic Parts” makes sounds when in sunlight.

Heap of Electronic Parts was a kind of hardware puzzle and certainly lives up to its name. It’s a bunch of parts soldered in a mystifying way to the backs of four old EPROMs — the chips with the little window through which UV is used to erase the contents. Assured that the unit really did have a function, [Maarten] eventually figured out that when placed in sunlight, the device ticks, buzzes, and squeals. [Jeroen] had figured out that the EPROMs could act like tiny solar cells when placed in sunlight, and together the four generate just enough power to drive an oscillator connected to a piezo speaker. It still chirps happily away, even today.

Hardware Tetris plays in a terminal window.

Hardware Tetris Unit was a black box intended to be plugged into a serial port. With a terminal opened using the correct serial port settings, a fully-functional Tetris game using ASCII-art graphics could be played. It was even self-powered from the serial port pins.

Inside Hardware Tetris is an AVR microcontroller with some level shifters, and the source code and schematics are available for download. 14 years later, computers no longer have hardware serial ports but [Maarten] says a USB-to-serial converter worked just fine and the device still functions perfectly.

There are a couple more devices documented on [Maarten]’s gifts page, including a Zork-inspired mini text adventure and a hardware board that does some trippy demos on an old Nokia color LCD.  [Maarten]’s friend [Jeroen Domburg] (aka Sprite_tm) had a hand in creating most of the gadgets, and he’s someone whose brilliant work we have had the good fortune to feature many times in the past.

Roll A Black Box For Your Wheels

Telemetric devices for vehicles, better known as black boxes, cracked the consumer scene 25 years ago with the premiere of OnStar. These days, you can get one for free from your insurance company if you want to try your luck at the discounts for safe driving game. But what if you wanted a black box just to mess around with that doesn’t share your driving data with the world? Just make one.

[TheForeignMan]’s DIY telematics box was designed to pull reports of the car’s RPM, speed, and throttle depression angle through the ODBII port. An ODBII-to-Bluetooth module sends the data to an Arduino Mega and logs it on an SD card along with latitude and longitude from a NEO-6M GPS module. Everything is powered by the car’s battery through a cigarette lighter-USB adapter.

He’s got everything tightly wrapped up inside a 3D printed box, which makes it pretty hard to retrieve the SD card. In the future, he’d like to send the data to a server instead to avoid accidentally dislodging a jumper wire.

If this one isn’t DIY enough for you to emulate, start by building your own CAN bus reader.

A Black Box For A Motorcycle


[Lukusz] has a new motorcycle – a Yamaha XJ6SA – and since it hasn’t been in an accident yet, he thought building a black box to record telemetry from the last 30 minutes of riding would be a good idea. While the project isn’t complete yet, he’s already reading data coming straight from the engine control unit.

After figuring out most of the pinout for his bike’s ECU connector, [Lukasz] found one wire that didn’t actually do anything. This was his ECU’s K line, a serial output that is able to relay the state of the gauges to external devices. The electronic spec of the K line is a bit weird, though, but luckily after finding a chip to convert the signal into something a logic analyzer can understand.

With a logic analyzer connected to the K line – and setting it to receive on at 16064 baud – [Lukasz] was able to get a whole lot of data directly from his bike. In the future he plans to pass data such as speed, indicator lights, RPMs, and the current gear to a Raspberry Pi for logging.