AI Mistakes Are Different, And That’s A Problem

January 25, 2025 by Donald Papp 59 Comments

People have been making mistakes — roughly the same ones — since forever, and we’ve spent about the same amount of time learning to detect and mitigate them. Artificial Intelligence (AI) systems make mistakes too, but [Bruce Schneier] and [Nathan E. Sanders] make the observation that, compared to humans, AI models make entirely different kinds of mistakes. We are perhaps less equipped to handle this unusual problem than we realize.

The basic idea is this: as humans we have tremendous experience making mistakes, and this has also given us a pretty good idea of what to expect our mistakes to look like, and how to deal with them. Humans tend to make mistakes at the edges of our knowledge, our mistakes tend to clump around the same things, we make more of them when bored or tired, and so on. We have as a result developed controls and systems of checks and balances to help reduce the frequency and limit the harm of our mistakes. But these controls don’t carry over to AI systems, because AI mistakes are pretty strange.

The mistakes of AI models (particularly Large Language Models) happen seemingly randomly and aren’t limited to particular topics or areas of knowledge. Models may unpredictably appear to lack common sense. As [Bruce] puts it, “A model might be equally likely to make a mistake on a calculus question as it is to propose that cabbages eat goats.” A slight re-wording of a question might be all it takes for a model to suddenly be confidently and utterly wrong about something it just a moment ago seemed to grasp completely. And speaking of confidence, AI mistakes aren’t accompanied by uncertainty. Of course humans are no strangers to being confidently wrong, but as a whole the sort of mistakes AI systems make aren’t the same kinds of mistakes we’re used to.

There are different ideas on how to deal with this, some of which researchers are (ahem) confidently undertaking. But for best results, we’ll need to invent new ways as well. The essay also appeared in IEEE Spectrum and isn’t terribly long, so take a few minutes to check it out and get some food for thought.

And remember, if preventing mistakes at all costs is the goal, that problem is already solved: GOODY-2 is undeniably the world’s safest AI.

Learning To Desolder Gracefully

July 24, 2021 by Elliot Williams 45 Comments

When you’re just learning to sketch, you use graphite. Why? It’s cheap, great at training you to recognize different shades, and most of all, it’s erasable. When you’re learning, you’re going to make mistakes, and un-making them is an important part of the game. Same goes for electronics, of course, so when you’re teaching someone to solder, don’t neglect teaching them to desolder.

We could argue all day about the best ways of pressing the molten-metal undo button, but the truth is that it’s horses for courses. I’ve had really good luck with solder braid and maybe a little heat gun to pull up reluctant SOIC surface-mount chips, but nothing beats a solder sucker for clearing out a few through-holes. (I haven’t tried the questionable, but time-tested practice of blasting the joint with compressed air.)

For bulk part removal, all you really have to do is heat the board up, and there’s plenty of ways to do that, ranging from fancy to foolish. Low-temperature alloys help out in really tough cases. And for removing rows of pinheaders, it can help to add more solder along the row until it’s one molten blob, and then tap the PCB and watch the part — and hot liquid metal! — just drop out.

But the bigger point is that an important step in learning a new technique is learning to undo your mistakes. It makes it all a lot less intimidating when you know that you can just pull out the solder braid and call “do-over”. And don’t forget the flux.

How Not To Build A Robotic Lawnmower

August 25, 2011 by Mike Nathan 26 Comments

[shadeydave] wanted to build his own Lawnbot, but he had no idea where to start. He purchased some DIY plans online which looked like they would get the job done, but then he strayed from the path in a big way and spent gobs of money in the process.

In his Instructable writeup, he details each misstep he made, explaining why his choices were bad as well as how much each mistake cost him. It sounds like pretty much everything that could go wrong did go wrong, from spending money on unnecessary microcontrollers to choosing the wrong wheels. Our favorite part is where he mentions that he couldn’t figure out how to create a “kill switch” for the Lawnbot in the event that his transmitter loses contact with the speedy whirling death machine.

[shadeydave] is well aware of how poorly his build went, and primarily wrote it up as a cautionary tale to others out there who might decide to take on a similar project. He says that the Lawnbot works for the most part, but with his newfound wisdom he will be revising the bot, having learned from his mistakes.

We actually like to see this kind of writeup as they can be quite beneficial to someone trying to put together a similar project. So if you have some major flubs under your belt, don’t be shy about digging them out and letting us know. As Thomas Edison said, “I have not failed. I have just found 10,000 ways that won’t work.”

Continue reading to see a quick video tour of [shadeydave’s] mostly working Lawnbot.

Continue reading “How Not To Build A Robotic Lawnmower” →