Linux Fu: Shell Script File Embedding

You need to package up a bunch of files, send them somewhere, and do something with them at the destination. It isn’t an uncommon scenario. The obvious answer is to create an archive — a zip or tar file, maybe — and include a shell script that you have to tell the user to run after unpacking.

That may be obvious, but it assumes a lot on the part of the remote user. They need to know how to unpack the file and they also need to know to run your magic script of commands after the unpack. However, you can easily create a shell script that contains a file — even an archive of many files — and then retrieve the file and act on it at run time. This is much simpler from the remote user’s point of view. You get one file, you execute it, and you are done.

In theory, this isn’t that hard to do, but there are a lot of details. Shell scripts are not compiled — at least, not typically — so the shell only reads what it needs to do the work. That means if your script is careful to exit, you can add as much “garbage” to the end of it as you like. The shell will never look at it, so it’s possible to store the payload there.

Continue reading “Linux Fu: Shell Script File Embedding”

Debugging For Sed — No Kidding

If you do much Linux shell scripting, you’ve probably encountered sed — the stream editor — in an example. Maybe you’ve even used it yourself. If all you want to do is substitute text, it is easy and efficient. But if you try to do really elaborate editing, it is often difficult to get things right. The syntax is cryptic and the documentation is lacking. But thanks to [SoptikHa2] you can now debug sed scripts with a text-based GUI debugger. Seriously.

According to the author, the program has several notable features:

  • Preview variable values, both of them!
  • See how will a substitute command affect pattern space before it runs
  • Step through sed script – both forward and backward!
  • Place breakpoints and examine program state
  • Hot reload and see what changes as you edit source code
  • Its name is a palindrome

There’s only one word for that last feature: wow.

Continue reading “Debugging For Sed — No Kidding”

Linux Fu: Leaning Down With Exec

Shell scripting is handy and with a shell like bash it is very capable, too. However, shell scripting isn’t always very efficient. Think about it. If you run grep or tr or sort to do some operation in a shell script, you are spawning a whole new process. That takes time and resources. But there are some answers to reducing — but not eliminating — the problem.

Have you ever written a program like this (in any language, but I’ll use C):

int foo(void)


You hope the compiler doesn’t write assembly code like this:


      call _bar

Most optimizers should pick up on the fact that you can convert a call like this to a jump and let the ret statement in _bar return to foo’s caller. However, shell scripts are not that smart. If you have a shell script called MungeData and it calls another program or shell script called PostProcess on its last line, then you will have at one time three processes in play: your original shell, the shell running MungeData, and either the PostProcess program or a shell running the script. Not to mention, the processes to do things inside post process. So what do you do?

Continue reading “Linux Fu: Leaning Down With Exec”

Script Your Way Out Of Video Editing Drudgery

[Victor Frost] has a deep voice and a fancy top of the line camera. While one would assume this to be a more than generous situation for life to put a person in; it’s got its own set of problems. Mainly that his fantastic fancy camera uses the most modern version of the popular h.264 encoding scheme, h.265. Gasp!

While that too seems like a pro, unfortunately h.265 doesn’t play as nice with his editing software. The solution seems easy, just transcode it and get on your way. However, when you start talking about transcoding 4K video from a top-of-the line source and retaining the quality. Well… It can bring a processor to its knees. Since he’d rather be playing overwatch than transcoding video on his main computer, he decided to offload and automate the drudgery to his spare.

That’s how the Ingest-a-Tron 9000 came into play. It uses a lot of open source software and, yes, windows batch files to take the files off his camera, process it on one computer, and dump it to another. Now he can game (or edit) while he waits. For those of us who are estranged from Linux thanks to our favorite software, it’s good to know that there are still ways to automate away the pain. Video after the break.

Continue reading “Script Your Way Out Of Video Editing Drudgery”

Shell Game

A lot of us spend a lot of time switching between Windows and Linux. Now that platforms like the Raspberry Pi are popular, that number is probably increasing every day. While I run Linux on nearly everything I own (with the exception of a laptop), my work computers mostly run Windows. The laptop is on Windows, too, because I got tired of trying to get all the fancy rotation sensors and pen features working properly under Linux.

What I hate most about Windows is how hard is it to see what’s going on under the hood. My HP laptop works with a cheap Dell active stylus. Sort of. It is great except around the screen edges where it goes wild. Calibration never works. On Linux, I could drill down to the lowest levels of the OS if I were so inclined. With Windows, it is just tough.

War is Shell

One place where Linux always used to have an advantage over DOS and Windows was the shell. There are lots of variations available under Linux, but bash seems to be the current pick for most people. If you want more power, you can move to some alternatives, but even bash is pretty powerful if you learn how to use it and have the right external programs (if you don’t believe it, check out this web server).

Continue reading “Shell Game”