AI Can Now Compress Text

April 28, 2024 by Jenny List 42 Comments

There are many claims in the air about the capabilities of AI systems, as the technology continues to ascend the dizzy heights of the hype cycle. Some of them are true, others stretch definitions a little, while yet more cross the line into the definitely bogus. [J] has one that is backed up by real code though, a compression scheme for text using an AI, and while there may be limitations in its approach, it demonstrates an interesting feature of large language models.

The compression works by assuming that for a sufficiently large model, it’s likely that many source texts will exist somewhere in the training. Using llama.cpp it’s possible to extract the tokenization information of a piece of text contained in its training data and store that as the compressed output. The decompressor can then use that tokenization data as a series of keys to reassemble the original from its training. We’re not AI experts but we are guessing that a source text which has little in common with any training text would fare badly, and we expect that the same model would have to be used on both compression and decompression. It remains a worthy technique though, and no doubt because it has AI pixie dust, somewhere there’s a hype-blinded venture capitalist who would pay millions for it. What a world we live in!

Oddly this isn’t the first time we’ve looked at AI text compression.

Raspinamp: It Really Replicates Questionable Activities Involving Llamas

March 7, 2024 by Bryan Cockfield 22 Comments

In the late 90s as MP3s and various file sharing platforms became more common, most of us were looking for better players than the default media players that came with our operating systems, if they were included at all. To avoid tragedies like Windows Media Center, plenty of us switched to Winamp instead, a much more customizable piece of software that helped pave the way for the digital music revolution of that era. Although there are new, official versions of Winamp currently available, nothing really tops the nostalgia of the original few releases of the software which this project faithfully replicates in handheld form.

The handheld music player uses a standard Raspberry Pi (in this case, a 3B) and a 3.5″ TFT touchscreen display, all enclosed in a clear plastic case. With all of the Pi configuration out of the way, including getting the touchscreen working properly, the software can be set up. It uses QMMP as a media player with a Winamp skin since QMMP works well on Linux systems with limited resources. After getting it installed there’s still some configuration to do to get the Pi to start it at boot and also to fit the player perfectly into the confines of the screen without any of the desktop showing around the edges.

Although it doesn’t use the original Winamp software directly, as that would involve a number of compatibility layers and/or legacy hardware at this point, we still think it’s a faithful recreation of how the original looked and felt on our Windows 98 machines. With a battery and a sizable SD card, this could have been the portable MP3 player many of us never knew we wanted until the iPod came out in the early 00s, and would certainly still work today for those of us not chained to a streaming service. A Raspberry Pi is not the only platform that can replicate the Winamp experience, though. This player does a similar job with the PyPortal instead.

Continue reading “Raspinamp: It Really Replicates Questionable Activities Involving Llamas” →

A Straightforward AI Voice Assistant, On A Pi

February 20, 2024 by Jenny List 6 Comments

With AI being all the rage at the moment it’s been somewhat annoying that using a large language model (LLM) without significant amounts of computing power meant surrendering to an online service run by a large company. But as happens with every technological innovation the state of the art has moved on, now to such an extent that a computer as small as a Raspberry Pi can join the fun. [Nick Bild] has one running on a Pi 4, and he’s gone further than just a chatbot by making into a voice assistant.

The brains of the operation is a Tinyllama LLM, packaged as a llamafile, which is to say an executable that provides about as easy a one-step access to a local LLM as it’s currently possible to get. The whisper voice recognition sytem provides a text transcript of the input prompt, while the eSpeak speech synthesizer creates a voice output for the result. There’s a brief demo video we’ve placed below the break, which shows it working, albeit slowly.

Perhaps the most important part of this project is that it’s easy to install and he’s provided full instructions in a GitHub repository. We know that the quality and speed of these models on commodity single board computers will only increase with time, so we’d rate this as an important step towards really good and cheap local LLMs. It may however be a while before it can help you make breakfast.

Continue reading “A Straightforward AI Voice Assistant, On A Pi” →

Using Local AI On The Command Line To Rename Images (And More)

December 29, 2023 by Donald Papp 36 Comments

We all have a folder full of images whose filenames resemble line noise. How about renaming those images with the help of a local LLM (large language model) executable on the command line? All that and more is showcased on [Justine Tunney]’s bash one-liners for LLMs, a showcase aimed at giving folks ideas and guidance on using a local (and private) LLM to do actual, useful work.

This is built out from the recent llamafile project, which turns LLMs into single-file executables. This not only makes them more portable and easier to distribute, but the executables are perfectly capable of being called from the command line and sending to standard output like any other UNIX tool. It’s simpler to version control the embedded LLM weights (and therefore their behavior) when it’s all part of the same file as well.

One such tool (the multi-modal LLaVA) is capable of interpreting image content. As an example, we can point it to a local image of the Jolly Wrencher logo using the following command:

llava-v1.5-7b-q4-main.llamafile --image logo.jpg --temp 0 -e -p '### User: The image has...\n### Assistant:'

Which produces the following response:

The image has a black background with a white skull and crossbones symbol.

With a different prompt (“What do you see?” instead of “The image has…”) the LLM even picks out the wrenches, but one can already see that the right pieces exist to do some useful work.

Check out [Justine]’s rename-pictures.sh script, which cleverly evaluates image filenames. If an image’s given filename already looks like readable English (also a job for a local LLM) the image is left alone. Otherwise, the picture is fed to an LLM whose output guides the generation of a new short and descriptive English filename in lowercase, with underscores for spaces.

What about the fact that LLM output isn’t entirely predictable? That’s easy to deal with. [Justine] suggests always calling these tools with the --temp 0 parameter. Setting the temperature to zero makes the model deterministic, ensuring that a same input always yields the same output.

There’s more neat examples on the Bash One-Liners for LLMs that demonstrate different ways to use a local LLM that lives in a single-file executable, so be sure to give it a look and see if you get any new ideas. After all, we have previously shown how automating tasks is almost always worth the time invested.

Mozilla Lets Folks Turn AI LLMs Into Single-File Executables

December 2, 2023 by Donald Papp 19 Comments

LLMs (Large Language Models) for local use are usually distributed as a set of weights in a multi-gigabyte file. These cannot be directly used on their own, which generally makes them harder to distribute and run compared to other software. A given model can also have undergone changes and tweaks, leading to different results if different versions are used.

To help with that, Mozilla’s innovation group have released llamafile, an open source method of turning a set of weights into a single binary that runs on six different OSes (macOS, Windows, Linux, FreeBSD, OpenBSD, and NetBSD) without needing to be installed. This makes it dramatically easier to distribute and run LLMs, as well as ensuring that a particular version of LLM remains consistent and reproducible, forever.

This wouldn’t be possible without the work of [Justine Tunney], creator of Cosmopolitan, a build-once-run-anywhere framework. The other main part is llama.cpp, and we’ve covered why it is such a big deal when it comes to running self-hosted LLMs.

There are some sample binaries available using the Mistral-7B, WizardCoder-Python-13B, and LLaVA 1.5 LLMs. Just keep in mind that if you’re on a Windows platform, only the LLaVA 1.5 will run, because it’s the only one that squeaks under the 4 GB limit on executable files that Windows has. If you run into issues, check out the gotchas list for troubleshooting tips.

When Is Open Source AI Not Open Source AI?

July 21, 2023 by Jenny List 18 Comments

The world of AI is abuzz, or at least parts of it are, at the news of Meta’s release of Llama 2. This is an AI text model which is thought to surpass ChatGPT in capabilities, and which the social media turned VR turned own all your things company wants you to know is open to all. That’s right, the code is open source and you can download the model, and Meta want you to feel warm and fuzzy about it. Unfortunately all is not as it seems, because of course the model isn’t open-source and is subject to a licensing restriction which makes it definitely not free of charge for larger users. This is of course disappointing to anyone hoping for an AI chatbot without restrictions, but we’re guessing Meta would prefer not to inadvertently enable a competitor.

Happily for the open source user large or small who isn’t afraid of a little work there’s an alternative in the form of OpenLLaMA, but we understand that won’t be for all users. Whichever LLM you use though, please don’t make the mistake of imagining that it possesses actual intelligence.

Thanks to the CoupledAI team for the tip!

Why LLaMa Is A Big Deal

March 22, 2023 by Matthew Carlson 27 Comments

You might have heard about LLaMa or maybe you haven’t. Either way, what’s the big deal? It’s just some AI thing. In a nutshell, LLaMa is important because it allows you to run large language models (LLM) like GPT-3 on commodity hardware. In many ways, this is a bit like Stable Diffusion, which similarly allowed normal folks to run image generation models on their own hardware with access to the underlying source code. We’ve discussed why Stable Diffusion matters and even talked about how it works.

LLaMa is a transformer language model from Facebook/Meta research, which is a collection of large models from 7 billion to 65 billion parameters trained on publicly available datasets. Their research paper showed that the 13B version outperformed GPT-3 in most benchmarks and LLama-65B is right up there with the best of them. LLaMa was unique as inference could be run on a single GPU due to some optimizations made to the transformer itself and the model being about 10x smaller. While Meta recommended that users have at least 10 GB of VRAM to run inference on the larger models, that’s a huge step from the 80 GB A100 cards that often run these models.

While this was an important step forward for the research community, it became a huge one for the hacker community when [Georgi Gerganov] rolled in. He released llama.cpp on GitHub, which runs the inference of a LLaMa model with 4-bit quantization. His code was focused on running LLaMa-7B on your Macbook, but we’ve seen versions running on smartphones and Raspberry Pis. There’s even a version written in Rust! A rough rule of thumb is anything with more than 4 GB of RAM can run LLaMa. Model weights are available through Meta with some rather strict terms, but they’ve been leaked online and can be found even in a pull request on the GitHub repo itself. Continue reading “Why LLaMa Is A Big Deal” →