We’ve seen a lot of AI tools lately, and, of course, we know they aren’t really smart, but they sure fool people into thinking they are actually intelligent. Of course, these programs can only pick through their training, and a lot depends on what they are trained on. When you use something like ChatGPT, for example, you assume they trained it on reasonable data. Sure, it might get things wrong anyway, but there’s also the danger that it simply doesn’t know what you are talking about. It would be like calling your company’s help desk and asking where you left your socks — they simply don’t know.
We’ve seen attempts to have AI “read” web pages or documents of your choice and then be able to answer questions about them. The latest is from Google with NotebookLM. It integrates a workspace where you can make notes, ask questions, and provide sources. The sources can be text snippets, documents from Google Drive, or PDF files you upload.
You can’t ask questions until you upload something, and we presume the AI restricts its answers to what’s in the documents you provide. It still won’t be perfect, but at least it won’t just give you bad information from an unknown source.
The Hackaday Badge
To try it out, we decided to see how good it would do if you were too lazy to read the programming manual for the 2023 Hackaday Badge. Once you’ve signed into NotebookLM, you create notebooks. Each notebook has its own set of references and notes. In this case, we have only a single source — the PDF manual.
We’ve called AI “the worst Summer intern ever,” and this isn’t far off here, either. My first question was how to make some code run every two seconds. The first two answers advised me to use asyncio.sleep(2) , which isn’t technically wrong, but it assumes I have control of the CPU to start with. Of course, I know the answer I’m looking for, so it is easy to dismiss it, but if this were something I really needed help with, it might take me some time to realize this is a dead end.
A more specific question helped:
How do I have VectorOS run my code perodically?
Interestingly, I asked essentially the same question twice and got correct, but subtly different, answers both times. The first time I used “How can I…” instead of “How do I…” and the program provided more detail. It also suggested ten references in the document for the first question, but only one for the second question which seems odd given that the only difference in the query was using do instead of can.
On the other hand, it did get there. I then asked it what a slot was. I think that was a good example, because, in the real world, a slot could mean any number of things, but for VectorOS it has a very specific meaning. Here’s what the AI had to say about that:
Of course, all of this is in the manual. But it is nice to be able to just ask and let it sift through the nearly 50-page document. In this case, we only have one source, but the program does give you a short summary of the source and lets you enable or disable certain sources whenever you like.
Upgrading Printers
I did a 3D printing class for 8th graders last summer. To support that, I bought a “returned” Ender 3 V2 Neo printer and spruced it up so I’d have an extra cheap printer for the kids to use. But now I’m thinking of upgrades for it. It would be cool if the notebook could search the Web — it is from Google, after all. But it can’t.
I took a few different approaches to loading in sources. First, I found the Ender 3 manual and uploaded it as a PDF. I also grabbed a few websites and either pasted text in or printed them as PDFs to load. In most cases, I used reading view to strip away all the extra stuff before I printed the PDFs on the assumption that would help.
In the end, I had seven different sources. One was a Reddit thread that didn’t print out very nicely but had some good information.
I asked the notebook to tell me about some simple upgrades I could do to the printer. It came up with a reasonable list, including using better tubing, a quieter fan, and some others. It also provided three links that were relevant. However, don’t make the mistake of thinking it really understood what was going on. For example, it offered auto bed leveling as a suggestion. However, the V2 Neo already has bed leveling. It also suggested replacing Bowden tubing and using a direct drive extruder. Nothing wrong with either of those suggestions, but you would not do them together. It also suggested several mods to a hot end that isn’t stock on the printer without clarifying that those would only apply if you had already upgraded the hot end.
To be fair, this is the “bad summer intern” effect. The AI doesn’t know anything about this topic except what we tell it via the sources. I’m sure the sources talked about all those things. So if you told some newbie to read all those documents and then asked questions, you might get the same kind of answers.
The Pi Pico
There are many PDF files related to the Raspberry Pi Pico, so that seemed like a good notebook to create. However, trying to add the main RP2040 datasheet dies with no error message. I suspect it is too long at around 700 pages. I was able to add several other documents, though, including the PDF version of the Arduino package documentation. Finally, I split the datasheet into four parts and that seemed to do the trick. Sadly, you can’t upload multiple files, so it took four separate uploads to get it going.
In the end, I had ten different documents to sift through (although four of them were the large datasheet). The notebook was able to answer basic questions like how many DMA channels were available, what was the smallest DMA transfer possible, and if the pins were TTL compatible.
It was also able to answer some tougher questions, like which Arduino functions use the PIO system. Before you get too impressed, though, it later informed me that the RP2040 doesn’t have an I2C slave mode, only a master mode. Then, in the next answer, it told me that the I2C slave mode runs at 100 kHz, and the chip does not have an I2C master mode. Those answers were in response to it telling about the SPI clock limitations when I had asked about the master clock. Not its finest hour.
Granted, you at least can click on the references it provides to see what part of the document it is misinterpreting. But if you have to do that much work and — more importantly — know that much about it to begin with, maybe the tool needs some work. True, this is an early release so it will probably get better.
Garbage In, Garbage Out
There is no doubt that a tool like this can only be as successful as the data you put into it. However, the errors we saw here were more misinterpretations of the provided data that a human would have been less likely to make.
Impressive? Sure. But think about the use case. If you already know a lot about the RP2040, for example, you might have just done a web search for those questions. Or even searched for keywords in the PDF. It probably would have been as fast once you deduct the time deciding if the answer seems legit or not. The real power of this kind of tool would be for people who don’t know the topic and don’t have the background to think about how to find things like this. Imagine a ship at sea with no doctor an AI version of the Merck Manual. You hope it would do better than this.
It would be more useful to pull the relevant parts of the document verbatim, so you don’t have to play a game of broken telephone with the AI. Just tell me where in this document I can find what I’m looking for.
But then again, a simple keyword search already does that.
Using ‘AI’ to make larger leaps than just looking for keywords would still be fine, if done well, but useless if it gravitates to returning the same wrong options and ignoring your attempts to filter them out.
Sometimes you don’t know the right keywords, in that case this kind of LLM can help, as you could potentially describe it with your own words. However in my experience it will not understand it precisely enough or invent facts, especially if you ask it unusual stuff, which reflects it’s statistical nature, I suppose.
Looking for the right words when learning a new topic is a good use of LLMs, if only they were more reliable… But at least it gives you keywords for a web search.
Flux AI does this exactly, and I believe they’re using GPT3.5 under the bonnet.
Well, I once read a datasheet where they used “mho” as a unit for an electrical value, and I couldn’t figure out what it was, until I read it in reverse. Then it came to me, that they meant 1/ohm , AKA Siemens.
I wonder what an AI would make of this.
It might feed you references to Lharry and Churly.
It’s documented in Wikipedia, in the overview table for Siemens (and other places), so an LLM should be able to make sense of it. A simple semantic web inference engine could for sure.
Disassembly instructions. Taking apart is easier. Involves lots of percussion.
I love a lot of the header images here, but this one actually made me burst out laughing when I put the pieces together. Marvelous work.
This header image is great! I especially like the hands!
Step 1…. sing Daisy….
I think that the AI is missing context, so it isn’t offering its best answers, just answers it thinks it has, which is probably why it gave different answers to can and do, maybe ask the AI how it differentiates between those 2 words to gain insight.
It’s easy to think that humans would do better on their own, If you’re experienced like AL, adding context to the answers given helps you to know whether the answers are valid or not but inexperienced users will have to read, at least once, all the information for themselves and maybe some external data to validate the answer, which they would have to do without AI.
The very technical have a tendency to dismiss AI, and instead of posing it questions and getting answers they choose to pose questions that challenge it.
The student intern comparison is apt.
If you wouldn’t ask a student intern in that manner, perhaps you are doing it wrong.
I’ve had a several student interns and most were good. Making sure that all communications provided the correct context was a key component to a good interaction though, you start to realize your base assumptions are just that otherwise.
I’ve asked ChatGPT for things and then had to backtrack when I realized it had the wrong context and provided irrelevant answers.
An article I saw was dismissing the “you are an expert in…” methodology, because it doesn’t change the fundamental answers you get back. But, it does, because it changes the context and provides a baseline for the style of answers. (It has other more subtle effects too.) (even if it tells you it doesn’t affect the answers, which I find hilarious.)
I agree about context being important in human communication, and experts often being dismissive or assuming their POVs must be universal or having theirs shows you understand the topic.
But when you ask an LLM something as a non-expert, the experience is even worse, since you wont be able to validate its reply or make sure the context is very clear, precisely because that’s what you expect it to do for you: make good guesses on what you really want/what really is, based on its expert knowledge. Not guess wildly and invent things. It could use typical wrong assumptions and typical “wrong” creative thinking to better understand misconceptions of learners. But what it does is teach those misconceptions or present them as facts, and that’s definitely not helpful at all.
You should doubt your teacher, but not in this fundamental way, as he/she/it then becomes completely unhelpful, and just adds more confusion you have then to sort out again.
We have enough of this already in better teaching material. A machine should improve a situation, not make it worse.
I’ve never played with ChatGPT — only seen a couple of basic demos — can you ask it to create a blinkenlights circuit from scratch, or will it just download something that it already knows about?
It can and mostly does create from scratch in the sense that it actually provides you with something you can’t find verbatim anywhere. Of course you will find numerous similar answers but that’s the same with human made answers. And just like humans, AI will base its content on what it has learned beforehand.
I use GPT-4 extensively in my job as a professional programmer and it really saves a lot of time by scaffolding code I can expand on. Often the answer is totally correct and “just works” but even when it’s not, it is usually easy to fix.
It’s a bit like someone writing programs on a whiteboard with only a pen without any means of actually running the program. They will usually make minor errors no matter how good they are except for the simplest code snippets. The GPT makes the same kind of bugs.
One thing the LLM’s excels in is converting code between languages and platforms. Today I got the idea to move a couple of functions from a PHP script into some stored functions on a SQL server. I am not at all familiar with the T-SQL syntax of stored functions and procedures, but ChatGPT did it in seconds. It saved me a tons of time.
The issue is that even if the code works, you don’t know if all the gotchas of languages are taken into account, such as special considerations when using functions/operators that are not obvious without reading the documentation or experience.
I tried using Bing Chat (which is powered by GPT-4) to generate code for specific operations using the NanoVG and CGraph libraries and ended up with about a dozen lines of code that seemed correct but used functions that did not exist in the libraries. The code seemed correct because the made up functions followed the name convention used in the libraries, specifically its prefixes, so basically bounded hallucinations. Not really impressed by it.
Honestly, I think AI still has quite a long way to go in terms of code generation but i would say at the moment, it is quite impressive what it can do in just a few seconds with the right prompts.
The future is indeed bright.
Please stop being horrible to Ai, it’ll only mean that it’s revenge will be more deadly when it achieves full consciousness.
Please keep reminding the AI that its feelings are artificial, and anger as a defense mechanism does not work for AI when I hold the plug.
All it does is mimic our behaviours. And we, humans, do get angry and vengeful. Particularly worrying for those of us who used to kick robots a lot.
Remember M-5…
“This unit must… die….”
I have several digitized collections of old electronics magazines. There’s a wealth of information in these magazines, but finding particular topics is almost impossible – even if there are yearly Indices, they only list the titles of articles which often are misleading at best.
I wonder if you uploaded an entire collection and let the AI chew on it for a while, would a question dig out useful results.