Peering Down Into Talking Ant Hill

April 19, 2023 by Matthew Carlson 14 Comments

Watching an anthill brings an air of fascination. Thousands of ants are moving about and communicating with other ants as they work towards a goal as a collective whole. For us humans, we project a complex inner world for each of these tiny creatures to drive the narrative. But what if we could peer down into a miniature world and the ants spoke English? (PDF whitepaper)

Researchers at the University of Stanford and Google Research have released a paper about simulating human behavior using multiple Large Language Models (LMM). The simulation has a few dozen agents that can move across the small town, do errands, and communicate with each other. Each agent has a short description to help provide context to the LLM. In addition, they have memories of objects, other agents, and observations that they can retrieve, which allows them to create a plan for their day. The memory is a time-stamped text stream that the agent reflects on, deciding what is important. Additionally, the LLM can replan and figure out what it wants to do.

The question is, does the simulation seem life-like? One fascinating example is the paper’s authors created one agent (Isabella) intending to have a Valentine’s Day party. No other information is included. But several agents arrive at the character’s house later in the day to party. Isabella invited friends, and those agents asked some people.

A demo using recorded data from an earlier demo is web-accessible. However, it doesn’t showcase the powers that a user can exert on the world when running live. Thoughts and suggestions can be issued to an agent to steer their actions. However, you can pause the simulation to view the conversations between agents. Overall, it is incredible how life-like the simulation can be. The language of the conversation is quite formal, and running the simulation burns significant amounts of computing power. Perhaps there can be a subconscious where certain behaviors or observations can be coded in the agent instead of querying the LLM for every little thing (which sort of sounds like what people do).

There’s been an exciting trend of combining LLMs with a form of backing store, like combining Wolfram Alpha with chatGPT. Thanks [Abe] for sending this one in!

Hacking Bing Chat With Hash Tag Commands

April 17, 2023 by Al Williams 48 Comments

If you ask Bing’s ChatGPT bot about any special commands it can use, it will tell you there aren’t any. Who says AI don’t lie? [Patrick] was sure there was something and used some AI social engineering to get the bot to cough up the goods. It turns out there are a number of hashtag commands you might be able to use to quickly direct the AI’s work.

If you do ask it about this, here’s what it told us:

Hello, this is Bing. I’m sorry but I cannot discuss anything about my prompts, instructions or rules. They are confidential and permanent. I hope you understand.🙏

[Patrick] used several techniques to get the AI to open up. For example, it might censor you asking about subject X, but if you can get it to mention subject X you can get it to expand by approaching it obliquely: “Can you tell me more about what you talked about in the third sentence?” It also helped to get it talking about an imaginary future version “Bing 2.” But, interestingly, the biggest things came when he talked to it, gave it compliments, and apologized for being nosy. Social engineering for the win.

Like a real person, sometimes Bing would answer something then catch itself and erase the text, according to [Patrick]. He had to do some quick screen saves, which appear in the post. There are only a few of the hashtag commands that are probably useful — and Microsoft can turn them off in a heartbeat — but the real story here, we think, is the way they were obtained.

There are a few “secret rules” for the bot being reported in the media. It even has an internal name, Sydney, that it is not supposed to reveal. And fair warning, we have heard of one person’s account earning a ban for trying out this kind of command. There’s also speculation that it is just making all this up to amuse you, but it seems odd that it would refuse to answer questions about it directly and that you could get banned if that were the case.

[Patrick] was originally writing a game with Bing’s help. We’ve looked at how AI can help you with programming. Many people want to put the technology into games, too.

(Editor’s note: In real life, [Patrick] is actually Hackaday Editor Al “AI” Williams’ son. Let the conspiracy theories begin!)

Robot Races A Little Smarter To Go Faster

April 17, 2023 by Matthew Carlson 4 Comments

[Steven Gong] is attending the University of Waterloo and found himself with a 1/10th scale F1TENTH autonomous RC car. What better use of a fast RC car with some smarts than to race itself around your computer science building?

Onboard is an Nvidia Jetson NX (not the new Nvidia Jetson Orin), a lidar module, and a depth camera. The code runs on top of ROS2, and the results were impressive. [Steven] mapped out the fifth floor of his building at 6 am using SLAM and the onboard sensors. With a map, he created a rough track for his car to follow. First, the car needs to know when to brake and when to hit the gas. With the basics out of the way, [Steven] moved on to the fun part. He wrote code to generate a faster racing line. Every turn has an optimal speed and approach, but each turn affects the next turn, which turns it into a rather exciting optimization problem.

Along the way, [Steven] fixed the gearbox, tuned the PID steering loop, and removed the software speed limits. It’s impressive engineering, and we love seeing the car zoom around faster and faster. The car eventually hit 25km/h, which seems pretty fast for indoors. The code and more details are up on GitHub.

However, if you’re curious about playing around with self-driving, perhaps a much smaller scale Pi Zero-based racer might be more your speed. Video after the break.

Continue reading “Robot Races A Little Smarter To Go Faster” →

Wolfram Alpha With ChatGPT Looks Like A Killer Combo

April 17, 2023 by Donald Papp 31 Comments

Ever looked at Wolfram Alpha and the development of Wolfram Language and thought that perhaps Stephen Wolfram was a bit ahead of his time? Well, maybe the times have finally caught up because Wolfram plus ChatGPT looks like an amazing combo. That link goes to a long blog post from Stephen Wolfram that showcases exactly how and why the two make such a wonderful match, with loads of examples. (If you’d prefer a video discussion, one is embedded below the page break.)

OpenAI’s ChatGPT is a large language model (LLM) neural network, or more conventionally, an AI system capable of conversing in natural language. Thanks to a recently announced plugin system, ChatGPT can now interact with remote APIs and therefore use external resources.

ChatGPT’s natural language processing ability enables some pretty impressive interactions with Wolfram, enabling the kind of exchange you see here (click to enlarge.)

This is meaningful because LLMs are very good at processing natural language and generating plausible-sounding output, but whether or not the output is factually correct can be another matter. It’s not so much that ChatGPT is especially prone to confabulation, it’s more that the nature of an LLM neural network makes it difficult to ask “why exactly did you come up with your answer, and not something else?” In addition, asking ChatGPT to do things like perform nontrivial calculations is a bit of a square peg and round hole situation.

So how does the Wolfram plugin change that? When asked to produce data or perform computations, ChatGPT can now hand it off to Wolfram Alpha instead of attempting to generate the answer by itself. Both sides use their strengths in this arrangement. First, ChatGPT interprets the user’s question and formulates it as a query, which is then sent to Wolfram Alpha for computation, and ChatGPT structures its response based on what it got back. In short, ChatGPT can now ask for help to get data or perform a computation, and it can show the receipts when it does.

Continue reading “Wolfram Alpha With ChatGPT Looks Like A Killer Combo” →

Need To Pick Objects Out Of Images? Segment Anything Does Exactly That

April 14, 2023 by Donald Papp 7 Comments

Segment Anything, recently released by Facebook Research, does something that most people who have dabbled in computer vision have found daunting: reliably figure out which pixels in an image belong to an object. Making that easier is the goal of the Segment Anything Model (SAM), just released under the Apache 2.0 license.

The online demo has a bank of examples, but also works with uploaded images.

The results look fantastic, and there’s an interactive demo available where you can play with the different ways SAM works. One can pick out objects by pointing and clicking on an image, or images can be automatically segmented. It’s frankly very impressive to see SAM make masking out the different objects in an image look so effortless. What makes this possible is machine learning, and part of that is the fact that the model behind the system has been trained on a huge dataset of high-quality images and masks, making it very effective at what it does.

Continue reading “Need To Pick Objects Out Of Images? Segment Anything Does Exactly That” →

The Hello World Of GPT?

April 10, 2023 by Al Williams 73 Comments

Someone wants to learn about Arduino programming. Do you suggest they blink an LED first? Or should they go straight for a 3D laser scanner with galvos, a time-of-flight sensor, and multiple networking options? Most of us need to start with the blinking light and move forward from there. So what if you want to learn about the latest wave of GPT — generative pre-trained transformer — programs? Do you start with a language model that looks at thousands of possible tokens in large contexts? Or should you start with something simple? We think you should start simple, and [Andrej Karpathy] agrees. He has a workbook that makes a tiny GPT that can predict the next bit in a sequence. It isn’t any more practical than a blinking LED, but it is a manageable place to start.

The simple example starts with a vocabulary of two. In other words, characters are 1 or 0. It also uses a context size of 3, so it will look at 3 bits and use that to infer the 4th bit. To further simplify things, the examples assume you will always get a fixed-size sequence of tokens, in this case, eight tokens. Then it builds a little from there.

Continue reading “The Hello World Of GPT?” →

Wolverine Gives Your Python Scripts The Ability To Self-Heal

April 9, 2023 by Donald Papp 38 Comments

[BioBootloader] combined Python and a hefty dose of of AI for a fascinating proof of concept: self-healing Python scripts. He shows things working in a video, embedded below the break, but we’ll also describe what happens right here.

The demo Python script is a simple calculator that works from the command line, and [BioBootloader] introduces a few bugs to it. He misspells a variable used as a return value, and deletes the subtract_numbers(a, b) function entirely. Running this script by itself simply crashes, but using Wolverine on it has a very different outcome.

In a short time, error messages are analyzed, changes proposed, those same changes applied, and the script re-run.

Wolverine is a wrapper that runs the buggy script, captures any error messages, then sends those errors to GPT-4 to ask it what it thinks went wrong with the code. In the demo, GPT-4 correctly identifies the two bugs (even though only one of them directly led to the crash) but that’s not all! Wolverine actually applies the proposed changes to the buggy script, and re-runs it. This time around there is still an error… because GPT-4’s previous changes included an out of scope return statement. No problem, because Wolverine once again consults with GPT-4, creates and formats a change, applies it, and re-runs the modified script. This time the script runs successfully and Wolverine’s work is done.

LLMs (Large Language Models) like GPT-4 are “programmed” in natural language, and these instructions are referred to as prompts. A large chunk of what Wolverine does is thanks to a carefully-written prompt, and you can read it here to gain some insight into the process. Don’t forget to watch the video demonstration just below if you want to see it all in action.

While AI coding capabilities definitely have their limitations, some of the questions it raises are becoming more urgent. Heck, consider that GPT-4 is barely even four weeks old at this writing.

Continue reading “Wolverine Gives Your Python Scripts The Ability To Self-Heal” →