Linux Fu: Failing Pipelines

July 25, 2024 by Al Williams 15 Comments

Bash is great for automating little tasks, but sometimes a little script you think will take a minute to write turns into a half hour or more. This is the story of one of those half-hour scripts.

I have too many 3D printers. In particular, I have three that are almost — but not exactly — the same, so each one has a slightly different build process when I want to update their firmware. In all fairness, one of those printers is heading out the door soon, but I’ll probably still wind up building firmware images for it.

My initial process was painful. I have a special directory with the four files needed to configure Marlin for each machine. I copy all four files and ask PlatformIO to perform the build. Usually, it succeeds and gives me a file that looks like firmware-yyyyddmmhhmm.bin or something like that.

The problem is that the build process doesn’t know which of the three machines is the target: Sulu, Checkov, or Fraiser. (Long story.) So, I manually look at the file name, copy it, and rename it. Of course, this is an error-prone process, and I’m basically lazy, so I decided to write a script to do it. I figured it would take just a minute to bundle up all the steps. I was wrong.

Continue reading “Linux Fu: Failing Pipelines” →

FLOSS Weekly Episode 790: Better Bash Scripting With Amber

July 3, 2024 by Jonathan Bennett 8 Comments

This week Jonathan Bennett and Dan Lynch chat with Paweł Karaś about Amber, a modern scripting language that compiles into a Bash script. Want to write scripts with built-in error handling, or prefer strongly typed languages? Amber may be for you!

Continue reading “FLOSS Weekly Episode 790: Better Bash Scripting With Amber” →

Preview Markdown In The Terminal With Bash

June 17, 2024 by Tom Nardi 12 Comments

Markdown has become an extremely popular way to document source code and other projects, thanks in no small part to how well web-based services like GitHub render it. Just sprinkle a few extra characters into a regular text file, and all of a sudden it looks like you know what you’re doing. Unfortunately, there are some places where markdown won’t actually render, and you’ll be stuck looking at those extra characters.

But thanks to MarCLIdown, the terminal doesn’t have to be one of those places. Written by [NihaAlGhul], this simple tool takes a given markdown file and spits out a fairly impressive rendering — and you don’t even need to have one of those fancy new GPU-accelerated terminals. Most impressively, the whole thing is implemented as a single Bash script.

Continue reading “Preview Markdown In The Terminal With Bash” →

Amber Compiles To Bash

May 22, 2024 by Al Williams 17 Comments

It certainly isn’t a new idea to compile a language into an intermediate language. The original C++ compiler outputs C code, for example. Enhanced versions of Fortran were often just conversions of new syntax to old syntax. Of course, it makes sense to output to some language that can run on lots of different platforms. So, using that logic, Amber makes perfect sense. It targets — no kidding — bash. You write with nice modern syntax and compile-time checks. The output is a bash script. Admittedly, sometimes a hard-to-read bash script, but still.

If you want to see the source code, it is available on GitHub. Since Windows doesn’t really support bash — if you don’t count things like Cygwin and WSL — Amber only officially supports Linux and MacOS. In addition to compiling files, Amber can also execute scripts directly which can be useful for a quick one-liner. If you use Visual Studio Code, you can find a syntax highlighter extension for Amber.

Continue reading “Amber Compiles To Bash” →

Is Your Mental Model Of Bash Pipelines Wrong?

March 28, 2024 by Donald Papp 11 Comments

[Michael Lynch] encountered a strange situation. Why was compiling then running his program nearly 10x faster than just running the program by itself? [Michael] ran into this issue while benchmarking a programming project, pared it down to its essentials for repeatability and analysis, and discovered it highlighted an incorrect mental model of how bash pipelines worked.

Here’s the situation. The first thing [Michael]’s pared-down program does is start a timer. Then it simply reads and counts some bytes from stdin, then prints out how long it took for that to happen. When running the test program in the following way, it takes about 13 microseconds.

$ echo '00010203040506070809' | xxd -r -p | zig build run -Doptimize=ReleaseFast
bytes: 10
execution time: 13.549µs

When running the (already-compiled) program directly, execution time swells to 162 microseconds.

$ echo '00010203040506070809' | xxd -r -p | ./zig-out/bin/count-bytes
bytes: 10
execution time: 162.195µs

Again, the only difference between zig build run and ./zig-out/bin/count-bytes is that the first compiles the code, then immediately runs it. The second simply runs the compiled program. Continue reading “Is Your Mental Model Of Bash Pipelines Wrong?” →

Using Local AI On The Command Line To Rename Images (And More)

December 29, 2023 by Donald Papp 36 Comments

We all have a folder full of images whose filenames resemble line noise. How about renaming those images with the help of a local LLM (large language model) executable on the command line? All that and more is showcased on [Justine Tunney]’s bash one-liners for LLMs, a showcase aimed at giving folks ideas and guidance on using a local (and private) LLM to do actual, useful work.

This is built out from the recent llamafile project, which turns LLMs into single-file executables. This not only makes them more portable and easier to distribute, but the executables are perfectly capable of being called from the command line and sending to standard output like any other UNIX tool. It’s simpler to version control the embedded LLM weights (and therefore their behavior) when it’s all part of the same file as well.

One such tool (the multi-modal LLaVA) is capable of interpreting image content. As an example, we can point it to a local image of the Jolly Wrencher logo using the following command:

llava-v1.5-7b-q4-main.llamafile --image logo.jpg --temp 0 -e -p '### User: The image has...\n### Assistant:'

Which produces the following response:

The image has a black background with a white skull and crossbones symbol.

With a different prompt (“What do you see?” instead of “The image has…”) the LLM even picks out the wrenches, but one can already see that the right pieces exist to do some useful work.

Check out [Justine]’s rename-pictures.sh script, which cleverly evaluates image filenames. If an image’s given filename already looks like readable English (also a job for a local LLM) the image is left alone. Otherwise, the picture is fed to an LLM whose output guides the generation of a new short and descriptive English filename in lowercase, with underscores for spaces.

What about the fact that LLM output isn’t entirely predictable? That’s easy to deal with. [Justine] suggests always calling these tools with the --temp 0 parameter. Setting the temperature to zero makes the model deterministic, ensuring that a same input always yields the same output.

There’s more neat examples on the Bash One-Liners for LLMs that demonstrate different ways to use a local LLM that lives in a single-file executable, so be sure to give it a look and see if you get any new ideas. After all, we have previously shown how automating tasks is almost always worth the time invested.

Suc Aims To Replace Slack In Five Lines Of Bash

July 10, 2023 by Bryan Cockfield 43 Comments

The design philosophy of Unix is fairly straightforward. Software should do one thing as simply as possible, and do that one thing only. As a design principle this is sound advice even well outside of the realm of Unix, and indeed software in general, but that doesn’t stop modern software packages from being too large for their own good. So, if you’re tired of bloated chat programs like Slack or Mattermost with their millions of lines of code, you might instead favor something like Simple Unix Chat (suc).

The idea is that suc can perform almost all modern chat functions in only five lines of Bash, supporting rich-text chat, file sharing, access control, and encryption. These five lines, though, only perform the core function of suc — which is to write text to a file on the system. Indeed, suc makes liberal use of plenty of other Unix services which do not add to the line counts, such as the use of SSH to handle authentication. It also relies on some other common Unix system features to handle things like ownership and access for the text files that host the text for the chat.

As channels are simply text files, it makes writing bots or other tools exceptionally simple. You can also easily pipe the output of commands directly into suc with one-liners that can do things like dump the output of make into a specific channel if compilation fails.

While it’s not likely that everyone will ditch tools like Slack to switch to something like this, it’s still an impressive demonstration of what can be done when designing around the Unix philosophy and taking advantage of system tools that already exist rather than reinventing the wheel and re-programming all of those tools into the application. Practices like this might decrease development time and increase the ease of developing cross-platform applications but they often also produce a less than desirable user experience.