Bridging A Gap Between LLMs And Programming With TypeChat

July 22, 2023 by Bryan Cockfield 33 Comments

By now, large language models (LLMs) like OpenAI’s ChatGPT are old news. While not perfect, they can assist with all kinds of tasks like creating efficient Excel spreadsheets, writing cover letters, asking for music references, and putting together functional computer programs in a variety of languages. One thing these LLMs don’t do yet though is integrate well with existing app interfaces. However, that’s where the TypeChat library comes in, bridging the gap between LLMs and programming.

TypeChat is an experimental MIT-licensed library from Microsoft which sits in between a user and a LLM and formats responses from the AI that are type-safe so that they can easily be plugged back in to the original interface. It does this by generating JSON responses based on user input, making it easier to take the user input directly, run it through the LLM, and then use the output directly in another piece of code. It can be used for things like prototyping prompts, validating responses, and handling errors. It’s also not limited to a single LLM and can be fairly easily modified to work with many of the existing models.

The software is still in its infancy but does hope to make it somewhat easier to work between user inputs within existing pieces of software and LLMs which have quickly become all the rage in the computer science world. We expect to see plenty more tools like this become available as more people take up using these new tools, which have plenty of applications beyond just writing code.

3D Design With Text-Based AI

May 12, 2023 by Bryan Cockfield 8 Comments

Generative AI is the new thing right now, proving to be a useful tool both for professional programmers, writers of high school essays and all kinds of other applications in between. It’s also been shown to be effective in generating images, as the DALL-E program has demonstrated with its impressive image-creating abilities. It should surprise no one as this type of AI continues to make in-roads into other areas, this time with a program from OpenAI called Shap-E which can render 3D images.

Like most of OpenAI’s offerings, this takes plain language as its input and can generate relatively simple 3D models with this text. The examples given by OpenAI include some bizarre models using text prompts such as a chair shaped like an avocado or an airplane that looks like a banana. It can generate textured meshes and neural radiance fields, both of which have various advantages when it comes to available computing power, training methods, and other considerations. The 3D models that it is able to generate have a Super Nintendo-style feel to them but we can only expect this technology to grow exponentially like other AI has been doing lately.

For those wondering about the name, it’s apparently a play on the 2D rendering program DALL-E which is itself a combination of the names of the famous robot WALL-E and the famous artist Salvador Dali. The Shap-E program is available for anyone to use from this GitHub page. Even though this code comes from OpenAI themselves, plenty are speculating that the AI revolution to come will largely come from open-source sources rather than OpenAI or Google, something for which the future is somewhat hazy.

Wolfram Alpha With ChatGPT Looks Like A Killer Combo

April 17, 2023 by Donald Papp 32 Comments

Ever looked at Wolfram Alpha and the development of Wolfram Language and thought that perhaps Stephen Wolfram was a bit ahead of his time? Well, maybe the times have finally caught up because Wolfram plus ChatGPT looks like an amazing combo. That link goes to a long blog post from Stephen Wolfram that showcases exactly how and why the two make such a wonderful match, with loads of examples. (If you’d prefer a video discussion, one is embedded below the page break.)

OpenAI’s ChatGPT is a large language model (LLM) neural network, or more conventionally, an AI system capable of conversing in natural language. Thanks to a recently announced plugin system, ChatGPT can now interact with remote APIs and therefore use external resources.

ChatGPT’s natural language processing ability enables some pretty impressive interactions with Wolfram, enabling the kind of exchange you see here (click to enlarge.)

This is meaningful because LLMs are very good at processing natural language and generating plausible-sounding output, but whether or not the output is factually correct can be another matter. It’s not so much that ChatGPT is especially prone to confabulation, it’s more that the nature of an LLM neural network makes it difficult to ask “why exactly did you come up with your answer, and not something else?” In addition, asking ChatGPT to do things like perform nontrivial calculations is a bit of a square peg and round hole situation.

So how does the Wolfram plugin change that? When asked to produce data or perform computations, ChatGPT can now hand it off to Wolfram Alpha instead of attempting to generate the answer by itself. Both sides use their strengths in this arrangement. First, ChatGPT interprets the user’s question and formulates it as a query, which is then sent to Wolfram Alpha for computation, and ChatGPT structures its response based on what it got back. In short, ChatGPT can now ask for help to get data or perform a computation, and it can show the receipts when it does.

Continue reading “Wolfram Alpha With ChatGPT Looks Like A Killer Combo” →

Tired Of Web Scraping? Make The AI Do It

April 9, 2023 by Donald Papp 19 Comments

[James Turk] has a novel approach to the problem of scraping web content in a structured way without needing to write the kind of page-specific code web scrapers usually have to deal with. How? Just enlist the help of a natural language AI. Scrapeghost relies on OpenAI’s GPT API to parse a web page’s content, pull out and classify any salient bits, and format it in a useful way.

What makes Scrapeghost different is how data gets organized. For example, when instantiating scrapeghost one defines the data one wishes to extract. For example:

from scrapeghost import SchemaScraper
scrape_legislators = SchemaScraper(
schema={
"name": "string",
"url": "url",
"district": "string",
"party": "string",
"photo_url": "url",
"offices": [{"name": "string", "address": "string", "phone": "string"}],
}
)

The kicker is that this format is entirely up to you! The GPT models are very, very good at processing natural language, and scrapeghost uses GPT to process the scraped data and find (using the example above) whatever looks like a name, district, party, photo, and office address and format it exactly as requested.

It’s an experimental tool and you’ll need an API key from OpenAI to use it, but it has useful features and is certainly a novel approach. There’s a tutorial and even a command-line interface, so check it out.

Let Machine Learning Code An Infinite Variety Of Pong Games

February 21, 2023 by Dan Maloney 7 Comments

In a very real way, Pong started the video game revolution. You wouldn’t have thought so at the time, with its simple gameplay, rudimentary controls, some very low-end sounds, and a cannibalized TV for a display, but the legendarily stuffed coinboxes tell the tale. Fast forward 50 years or so, and Pong has been largely reduced to a programmer’s exercise to see how few lines of code can stand in for what [Ted Dabney] and [Allan Alcorn] accomplished. But now even that’s too much, as OpenAI Codex can generate a playable Pong from just a few prompts, at least most of the time. Continue reading “Let Machine Learning Code An Infinite Variety Of Pong Games” →

With ChatGPT, Game NPCs Get A Lot More Interesting

February 8, 2023 by Donald Papp 24 Comments

Not only is AI-driven natural language processing a thing now, but you can even select from a number of different offerings, each optimized for different tasks. It took very little time for [Bloc] to mod a computer game to allow the player to converse naturally with non-player characters (NPCs) by hooking it into ChatGPT, a large language model AI optimized for conversational communication.

If you can look past the painfully-long loading times, even buying grain (7:36) gains a new layer of interactivity.

[Bloc] modified the game Mount & Blade II: Bannerlord to reject traditional dialogue trees and instead accept free-form text inputs, using ChatGPT on the back end to create more natural dialogue interactions with NPCs. This is a refinement of an earlier mod [Bloc] made and shared, so what you see in the video below is quite a bit more than a proof of concept. The NPCs communicate as though they are aware of surrounding events and conditions in the game world, are generally less forthcoming when talking to strangers, and the new system can interact with game mechanics and elements such as money, quests, and hirelings.

Starting around 1:08 into the video, [Bloc] talks to a peasant about some bandits harassing the community, and from there demonstrates hiring some locals and haggling over prices before heading out to deal with the bandits.

The downside is that ChatGPT is currently amazingly popular. As a result, [Bloc]’s mod is stuck using an overloaded service which means some painfully-long load times between each exchange. But if you can look past that, it’s a pretty fascinating demonstration of what’s possible by gluing two systems together with a mod and some clever coding.

Take a few minutes to check out the video, embedded below. And if you’re more of a tabletop gamer? Let us remind you that it might be fun to try replacing your DM with ChatGPT.

Continue reading “With ChatGPT, Game NPCs Get A Lot More Interesting” →

Giving An Old Typewriter A Mind Of Its Own With GPT-3

January 15, 2023 by Dan Maloney 27 Comments

There was an all-too-brief period in history where typewriters went from clunky, purely mechanical beasts to streamlined, portable electromechanical devices. But when the 80s came around and the PC revolution started, the typewriting was on the wall for these machines, and by the 90s everyone had a PC, a printer, and Microsoft Word. And thus the little daisy-wheel typewriters began to populate thrift shops all over the world.

That’s fine with us, because it gave [Arvind Sanjeev] a chance to build “Ghostwriter”, an AI-powered automatic typewriter. The donor machine was a clapped-out Brother electronic typewriter, which needed a bit of TLC to bring it back to working condition. From there, [Arvind] worked out the keyboard matrix and programmed an Arduino to drive the typewriter, both read and write. A Raspberry Pi running the OpenAI Python API for GPT-3 talks to the Arduino over serial, which basically means you can enter a GPT writing prompt with the keyboard and have the machine spit out a dead-tree version of the results.

To mix things up a bit, [Arvind] added a pair of pots to control the creativity and length of the response, plus an OLED screen which seems only to provide some cute animations, which we don’t hate. We also don’t hate the new paint job the typewriter got, but the jury is still out on the “poetry” that it typed up. Eye of the beholder, we suppose.

Whatever you think of GPT’s capabilities, this is still a neat build and a nice reuse of otherwise dead-end electronics. Need a bit more help building natural language AI into your next project? Our own [Donald Papp] will get you up to speed on that.

Continue reading “Giving An Old Typewriter A Mind Of Its Own With GPT-3” →