Linux Fu: Global Search And Replace With Ripgrep

October 14, 2020 by Al Williams 9 Comments

If you are even a casual Linux user, you probably know how to use grep. Even if you aren’t a regular expression guru, it is easy to use grep to search for lines in a file that match anything from simple strings to complex patterns. Of course, grep is fine for looking, but what if you want to find things and change them. Maybe you want to change each instance of “HackADay” to “Hackaday,” for example. You might use sed, but it is somewhat hard to use. You could use awk, but as a general-purpose language, it seems a bit of overkill for such a simple and common task. That’s the idea behind ripgrep which actually has the command name rg. Using rg, you can do things that grep can do using more modern regular expressions and also do replacements.

A Note on Installing Ripgrep

Your best bet is to get ripgrep from your repositories. When I tried running KDE Neon, it helpfully told me that I could install a version using apt or take a Snap version that was newer. I usually hate installing a snap, but I did anyway. It informed me that I had to add –classic to the install line because ripgrep could affect files outside the Snap sandbox. Since the whole purpose of the program is to change files, I didn’t think that was too surprising, so I did the install.

Continue reading “Linux Fu: Global Search And Replace With Ripgrep” →

Linux Fu: The Linux Android Convergence

October 7, 2020 by Al Williams 63 Comments

The Android phone that you carry in your pocket is basically a small computer running Linux. So why is it so hard to get to a usable Linux environment on your phone? If you could run Linux, you could turn your cell phone into an ultra-portable laptop replacement.

Of course, the obvious approach is just to root the phone and clean-slate install a Linux distribution on it. That’s pretty extreme and, honestly, you would probably lose a lot of phone function unless you go with a Linux-specific phone like the PinePhone. However, using an installer called AnLinux, along with a terminal program and a VNC client, you can get a workable setup without nuking your phone’s OS, or even having root access. Let’s see what we can do. Continue reading “Linux Fu: The Linux Android Convergence” →

Should You Build For Windows, Mac, IOS, Android, Or Linux? Yes!

October 1, 2020 by Al Williams 36 Comments

The holy grail of computer languages is to write code once and have it deploy effortlessly everywhere. Java likes to take credit for the idea, but UCSD P-Code was way before that and you could argue that mainframes had I/O abstraction like Fortran unit numbers even earlier. More modern efforts include Qt, GTK, and other things. Naturally, all of these fall short in some way. Now Google enters the fray with Flutter.

Flutter isn’t new, but in the past, it only handled Android and iOS. Now it can target desktop platforms and can even produce JavaScript. We haven’t played with the system enough to say how successful it is, but you can try it in your browser if you want some first-hand experience.

Continue reading “Should You Build For Windows, Mac, IOS, Android, Or Linux? Yes!” →

Linux-Fu: Making AWK A Bit Easier

September 29, 2020 by Al Williams 18 Comments

awk is a kind of Swiss Army knife for text files. However, some of its limitations are often a bit annoying. I’ve used a simple set of functions to make awk a bit better, although I will warn you: it does require GNU extensions to awk. That is, you must use gawk and not other versions. Your system probably maps /usr/bin/awk to something and that something might be gawk. But it could also be mawk or some other flavor. If you use a Debian-based distro, update-alternatives is your friend here. But for the purposes of this post, I’m going to assume you are using gawk.

By the end of the post, you’ll see how to use my awk add-on functions to split up a line into fields even when there is no single character to separate all fields. In addition, you’ll be able to refer to the fields using names you decide. You won’t have to remember that $2 is the time field. You’ll say Fields_fields["time"] instead.

The Problem

awk does a lot of common work for you when you use it to process text files. It reads files a record at a time. Normally, a record is a single line. Then it splits the line on fields using whitespace, or some other choice of field separators. You can write code that manipulates the line or individual fields. This default behavior is great, especially since you can change the end of record character and the field separator. A surprising number of files fit this sort of format.

Until, of course, they don’t. If you have data coming from a data logging instrument or some database, it could be formatted in a variety of ways. Some fields might have structured data with a variety of separators. This isn’t a deal-breaker. Since you can get at the whole line, you can do almost anything you want, but the logic is harder and the whole point to using awk is to make things easier.

For example, suppose you had a file from a data recorder that had an eight-digit serial number, followed by a six-character tag, and then two floating point numbers separated by colons. The pattern might look like

^([0-9]{8})([a-zA-Z0-9]{6})([-+.0-9]+),([-+.0-9]+)$

This would be hard to handle with the conventional field splitting and you’d normally just write code to split everything apart.

Continue reading “Linux-Fu: Making AWK A Bit Easier” →

Linux Fu: Simple SSH File Sharing

September 21, 2020 by Al Williams 28 Comments

If you have more than one Linux computer, you probably use ssh all the time. It is a great tool, but I’ve always found one thing about it strange. Despite having file transfer capabilities in the form of scp and sftp, there is no way to move a file back or forth between the local and remote hosts without starting a new program on the local machine or logging in from the remote machine back to the local machine.

That last bit is a real problem since you often access a server from behind a firewall or a NAT router with an ephemeral IP address, so it can’t reconnect to you anyway. It would be nice to hit the escape character, select a local or remote file, and teleport it across the interface, all from inside a single ssh session.

I didn’t quite get to that goal, but I did get pretty close. I’ll show you a script that can automatically mount a remote directory on the local machine. You’ll need sshfs on the local machine, but no changes on the remote machine where you may not be able to install software. With a little more work, and if your client has an ssh server running, you can mount a local directory on the remote machine, too. You won’t need to worry about your IP address or port blocking. If you can log into the remote machine, you are good.

Combined, this got me me very close to my goal. I can be working in a shell on either side and have access to read or write files on the other side. I just have to set it up carefully. Continue reading “Linux Fu: Simple SSH File Sharing” →

Linux Fu: Literate Regular Expressions

September 11, 2020 by Al Williams 20 Comments

Regular expressions — the things you feed to programs like grep — are a bit like riding a bike. It seems impossible until you learn to do it, and then it’s easy. Part of their bad reputation is because they use a very concise and abbreviated syntax that alarms people. To help people who don’t use regular expressions every day, I created a tool that lets you write them in something a little closer to plain English. Actually, I’ve written several versions of this over the years, but this incarnation that targets grep is the latest. Unlike some previous versions, this time I did it all using Bash.

Those who don’t know regular expressions might freak out when they see something like:

[0-9]{5}(-[0-9]{4})?

How long does it take to figure out what that does? What if you could write that in a more literate way? For example:

digit repeat 5 \

start_group \

   - digit repeat 4 \

end_group optional

Not as fast to type, sure. But you can probably deduce what it does: it reads US Zipcodes.

I’ve found that some of the most popular tools I’ve created over the years are ones that I don’t need myself. I’m sure you’ve had that experience, too. You know how to operate a computer, but you create a menu system for people who don’t and they love it. That’s how it is with this tool. You might not need it, but there’s a good chance you know someone who does. Along the way, the code uses some interesting features of Bash, so even if you don’t want to be verbose with your regular expressions, you might pick up a trick or two.

Continue reading “Linux Fu: Literate Regular Expressions” →

A Free Software OS For The ReMarkable E-Paper Tablet

September 6, 2020 by Tom Nardi 83 Comments

If you’re looking to rid your day to day life of dead trees, there’s a good chance you’ve already heard of the reMarkable tablet. The sleek device aims to replace the traditional notebook. To that end, remarkable was designed to mimic the feeling of writing on actual paper as closely as possible. But like so many modern gadgets, it’s unfortunately encumbered by proprietary code with a dash of vendor lock-in. Or at least, it was.

[Davis Remmel] has been hard at work porting Parabola, a completely free and open source GNU/Linux distribution, to the reMarkable. Developers will appreciate the opportunity to audit and modify the OS, but even from an end-user perspective, Parabola greatly opens up what you can do on the device. Before you were limited to a tablet UI and a select number of applications, but with this replacement OS installed, you’ll have a full-blown Linux desktop to play with.

You still won’t be watching videos or gaming on the reMarkable (though technically, you would be able to), but you could certainly use it to read and edit documents the original OS didn’t support. You could even use it for light software development. Since USB serial adapters are supported, microcontroller work isn’t out of the question either. All while reaping the considerable benefits of electronic paper.

The only downside is that the WiFi hardware is not currently supported as it requires proprietary firmware to operate. No word on whether or not [Davis] is willing to make some concession there for users who aren’t quite so strict about their software freedoms.

We’ve been waiting patiently for the electronic paper revolution to do more than replace paperbacks with Kindles, and devices like the reMarkable seem to be finally moving us in the right direction. Thankfully, projects that aim to bring free and open source software to these devices mean we won’t necessarily have to let Big Brother snoop through our files in the process.