Measuring The Impact Of LLMs On Experienced Developer Productivity

Recently AI risk and benefit evaluation company METR ran a randomized control test (RCT) on a gaggle of experienced open source developers to gain objective data on how the use of LLMs affects their productivity. Their findings were that using LLM-based tools like Cursor Pro with Claude 3.5/3.7 Sonnet reduced productivity by about 19%, with the full study by [Joel Becker] et al. available as PDF.

This study was also intended to establish a methodology to assess the impact from introducing LLM-based tools in software development. In the RCT, 16 experienced open source software developers were given 246 tasks, after which their effective performance was evaluated.

A large focus of the methodology was on creating realistic scenarios instead of using canned benchmarks. This included adding features to code, bug fixes and refactoring, much as they would do in the work on their respective open source projects. The observed increase in the time it took to complete tasks with the LLM’s assistance was found to be likely due to a range of factors, including over-optimism about the LLM tool capabilities, LLMs interfering with existing knowledge on the codebase, poor LLM performance on large codebases, low reliability of the generated code and the LLM doing very poorly on using tacit knowledge and context.

Although METR suggests that this poor showing may improve over time, it seems fair to argue whether LLM coding tools are at all a useful coding partner.

DIY X-Rays Made Easy

Who doesn’t want an X-ray machine? But you need a special tube and super high voltage, right? [Project 326] says no, and produces a USB-powered device that uses a tube you can pick up two for a dollar. You might guess the machine doesn’t generate X-rays with a lot of energy, and you’d be right. But you can make up for it with long exposure times. Check out the video below, with host [Posh Arthur].

The video admits there are limitations, of course. We were somewhat sad that [Project 326] elected not to share the exact parts list and 3D printed files because in the unlikely event someone managed to hurt themselves with it, there could be a hysterical reaction. We agreed, though, that if you are smart enough to handle this, you’ll be smart enough to figure out how to duplicate it — it doesn’t look that hard, and there are plenty of not-so-subtle clues in the video.

Continue reading “DIY X-Rays Made Easy”

Four brown perf board circuits are visible in the foreground, each populated with many large DIP integrated circuits. The boards are connected with grey ribbon cable. Behind the boards a vacuum fluorescent display shows the words “DIY CPU.”

Designing A CPU With Only Memory Chips

Building a simple 8-bit computer is a great way to understand computing fundamentals, but there’s only so much you can learn by building a system around an existing processor. If you want to learn more, you’ll have to go further and build the CPU yourself, as [MINT] demonstrated with his EPROMINT project (video in Polish, but with English subtitles).

The CPU began when [MINT] began experimenting with uses for his collection of old memory chips, and quickly realized that they could do quite a bit more than store data. After building a development board for single-chip based programmable logic, he decided to build a full CPU out of (E)EPROMs. The resulting circuit spans four large pieces of perfboard, weighs in at over half a kilogram, and took several weeks of soldering to create. Continue reading “Designing A CPU With Only Memory Chips”

Presenter holds an induction lamp bulb

An Induction Lamp Made On The Same Principle As Ordinary Fluorescent Lamp

Over on YouTube, [Technology Connections] has a new video: Induction lamps: fluorescent lighting’s final form.

This video is about a wireless fluorescent light which uses induction to transfer power from the electrical system into the lamp. As this lamp doesn’t require wiring it is not prone to “sputtering” as typical fluorescent lights are, thus improving the working life by an order of magnitude. As explained in the video sputtering is the process where the electrodes in a typical fluorescent lamp lose their material over time until they lose their ability to emit electrons at all.

This particular lamp has a power rating of 200 W and light output of 16,000 lumens, which is quite good. But the truly remarkable thing about this type of lighting is its service life. As the lamp is simply a phosphor-coated tube filled with argon gas and a pellet of mercury amalgam it has a theoretically unlimited lifespan. Or let’s call it 23 years.

Given that the service life is so good, why don’t we see induction lamps everywhere? The answer is that the electronics to support them are very expensive, and these days LED lighting has trounced every lighting technology that we’ve ever made in terms of energy efficiency, quality of light, and so on. So induction lamps are obsolete before they ever had their day. Still pretty interesting technology though!

Continue reading “An Induction Lamp Made On The Same Principle As Ordinary Fluorescent Lamp”

Dearest C++, Let Me Count The Ways I Love/Hate Thee

My first encounter with C++ was way back in the 1990s, when it was one of the Real Programming Languages™ that I sometimes heard about as I was still splashing about in the kiddie pool with Visual Basic, PHP and JavaScript. The first formally standardized version of C++ is the ISO 1998 standard, but it had been making headways as a ‘better C’ for decades at that point since Bjarne Stroustrup added that increment operator to C in 1979 and released C++ to the public in 1985.

Why did I pick C++ as my primary programming language? Mainly because it was well supported and with free tooling: a free Borland compiler or g++ on the GCC side. Alternatives like VB, Java, and D felt far too niche compared to established languages, while C++ gave you access to the lingua franca of C while adding many modern features like OOP and a more streamlined syntax in addition to the Standard Template Library (STL) with gobs of useful building blocks.

Years later, as a grizzled senior C++ developer, I have come to embrace the notion that being good at a programming language also means having strong opinions on all that is wrong with the language. True to form, while C++ has many good points, there are still major warts and many heavily neglected aspects that get me and other C++ developers riled up.

Continue reading “Dearest C++, Let Me Count The Ways I Love/Hate Thee”

Hackaday Podcast Episode 328: Benchies, Beanies, And Back To The Future

This week, Hackaday’s Elliot Williams and Kristina Panos joined forces to bring you the latest news, mystery sound, and of course, a big bunch of hacks from the previous week.

In Hackaday news, the One Hertz Challenge ticks on. You have until Tuesday, August 19th to show us what you’ve got, so head over to Hackaday.IO and get started now! In other news, we’ve just wrapped the call for Supercon proposals, so you can probably expect to see tickets for sale fairly soon.

On What’s That Sound, Kristina actually got this one with some prodding. Congratulations to [Alex] who knew exactly what it was and wins a limited edition Hackaday Podcast t-shirt!

After that, it’s on to the hacks and such, beginning with a ridiculously fast Benchy. We take a look at a bunch of awesome 3D prints a PEZ blaster and a cowbell that rings true. Then we explore chisanbop, which is not actually K-Pop for toddlers, as well as a couple of clocks. Finally, we talk a bit about dithering before taking a look at the top tech of 1985 as shown in Back to the Future (1985).

Check out the links below if you want to follow along, and as always, tell us what you think about this episode in the comments!

Download in DRM-free MP3 and savor at your leisure.

Continue reading “Hackaday Podcast Episode 328: Benchies, Beanies, And Back To The Future

PlayStation Case Mod Hides Gamer Shame

[Zac] of Zac Builds has a shameful secret: he, a fully grown man, plays video games. Shocking, we know, but such people do exist in our society. After being rightfully laughed out of the family living room, [Zac] relocated his indecent activities to his office, but he knew that was not enough. Someone might enter, might see his secret shame: his PlayStation 5. He decided the only solution was to tear the game console apart, and rebuild it inside of his desk.

All sarcasm aside, it’s hard to argue that [Zac]’s handmade wooden desk doesn’t look better than the stock PS5, even if you’re not one of the people who disliked Sony’s styling this generation. The desk also contains his PC, a project we seem to have somehow missed; the two machines live in adjacent drawers.

While aesthetics are a big motivator behind this case mod, [Zac] also takes the time to improve on Sony’s work: the noisy stock fan is replaced by three silent-running Noctua case fans; the easy-to-confuse power and eject buttons are relocated and differentiated; and the Blu-ray drive gets a proper affordance so he’ll never miss the slot again. An NVMe SSD finishes off the upgrades.

Aside from the woodworking to create the drawer, this project relies mostly on 3D printing for custom mounts and baffles to hold the PS5’s parts and direct airflow where it needs to go. This was made much, much easier for [Zac] via the use of a 3D scanner. If you haven’t used one, this project demonstrates how handy they can be — and also some of the limitations, as the structured-light device (a Creality Raptor) had trouble with the shinier parts of the build. Dealing with that trouble still saved [Zac] a lot of time and effort compared to measuring everything.

While we missed [Zac]’s desk build, we’ve seen his work before: everything from a modernized iPod to wooden sound diffusion panels.

Continue reading “PlayStation Case Mod Hides Gamer Shame”