Linux Fu: Improving FTP

FTP isn’t exactly cutting-edge technology. These days, if you control both ends of a connection, you’re probably using scp, SFTP, rsync, or something even fancier. But FTP refuses to die, especially if you are perusing old public FTP servers or talking to retrocomputers. Every now and then, you still need an FTP client. Naturally, there are plenty of graphical clients. But some of us would rather stay at the command line. You could just type ftp, of course. It works, and if you haven’t used it lately, it is probably better than you remember. However, I’ve long been a fan of NcFTP. While some other FTP clients have caught up, it still has unique features that make FTP a lot more productive.

Not Your Father’s FTP

Before maligning the standard ftp command, though, we should point out that it probably isn’t the FTP client you remember from 30 years ago. For example, on openSUSE Tumbleweed, /usr/bin/ftp is really tnftp, a portable version of NetBSD’s enhanced FTP client. Debian uses it too; the ftp package in both Bookworm and Trixie leads you to tnftp. Since current Raspberry Pi OS is based on Debian Trixie, you’ll encounter tnftp there, too. That’s significant because tnftp has already fixed many of the irritations you might associate with old-fashioned FTP.

You get command-line editing, history, and filename completion, things that are also in ncftp. Both understand passive FTP and IPv6. The tnftp client can also retrieve HTTP, HTTPS, and file: URLs, so commands such as:

ftp https://example.com/something.tar.gz

aren’t necessarily typos, although ncftp lacks this ability. But ncftp does have some killer features.

Remember Me?

One of NcFTP’s nicest creature comforts is bookmarks. Connect to a machine, move to a useful directory, and save it:

ncftp /pub/micros> bookmark oldstuff

Then later you can simply type:

ncftp oldstuff

The bookmark can remember more than just the hostname, making frequently used FTP sites feel much more like named resources than anonymous servers you repeatedly have to navigate.

NcFTP also maintains a cache of remote directory listings. If you’ve ever used FTP over a slow link, you know how annoying it is to ask for the same directory listing over and over. NcFTP can often work from what it already knows instead. Neither feature sounds earth-shattering, but together they make an interactive FTP session considerably more pleasant.

Get All The Things

Another difference becomes obvious when you want an entire directory. NcFTP supports recursive transfers:

get -R foo

or:

put -R foo

That seems obvious if you’re accustomed to modern tools, but traditional FTP is fundamentally organized around transferring individual files. NcFTP does the tedious directory walking for you. It also handles resuming interrupted transfers more naturally, something particularly welcome when the file in question is a multi-gigabyte disk image rather than README.TXT. With tnftp, you have to explicitly ask to resume an interrupted file. NcFTP will detect it and, depending on configuration, either resume or, at least, offer to resume the transfer.

Go Away, I’m Busy

NcFTP also has a clever background-transfer system. Commands such as:

bgget giant-file.iso

Hand a transfer to NcFTP’s spooler rather than tying up your interactive session. There are corresponding facilities for uploads. That’s an interesting distinction from simply detaching a shell command. NcFTP knows that this is a transfer job and maintains a queue of FTP work that can be retried and processed independently.

Shell Games

But perhaps the biggest reason to know about NcFTP is that NcFTP isn’t just one program. The package includes commands such as ncftpget, ncftpput, and ncftpls. These perform FTP operations directly from the Unix shell without starting an interactive FTP command interpreter. For example:

ncftpget ftp.example.com /tmp /pub/widget.bin

or:

ncftpput ftp.example.com /incoming widget.bin

This is much nicer in a script than sending commands to ftp using, for example, a here document and automating login with .netrc. For example:

ftp <<EOF
open ftp.example.com
cd incoming
put widget.bin
quit
EOF

Sure, it works, but any time you send input to an interactive program it is, at best, messy. The ncftpput program expresses what you actually wanted to do in the first place: put this file there. That’s much more Unix-like.

Don’t Do This At Home

None of these conveniences change FTP’s fundamental problem: ordinary FTP is not secure. Usernames, passwords, and data can travel without encryption. If you’re designing a new system and control both ends, you usually have much better choices. But sometimes you don’t control both ends. If FTP is something you run into, ncftp is worth knowing about. Bookmarks, cached directories, recursive and background transfers, and especially the script-friendly companion commands turn an antique protocol into something that feels surprisingly at home on a modern Unix command line.

Of course, just as you can use FUSE to mount an ssh server, you can use ftpfs, to make a remote server look like part of your file system. You never know when FTP is going to crop up.

Linux Fu: Heads Or Tails For VPN

If you’ve done much networking, you surely know the frustration of trying to connect to something, say a Raspberry Pi, that lives behind your consumer router. There are a number of solutions for this, ranging from opening ports on your router along with dynamic DNS. Or, you can operate a VPN server on your network. Modern Linux has a facility called Wireguard that lets you create secure network tunnels very easily, but it is a little difficult to set up. But there are tools like Tailscale that can do most of the work for you. There’s only one problem: Tailscale is sorta-kinda free, but not really. But it turns out, you can build your own Tailscale network, and it is easier than you might imagine.

In all fairness, Tailscale’s free tier is good and recently got even more generous, allowing unlimited nodes and up to six users. That’s plenty for most hackers. However, as we’ve seen before, what they can give they can also take away. Besides, there are some extra services you still have to pay for if you want them, but overall, the free tier is more than enough for most people.

On the other hand, no matter how great the free tier may be, some people don’t want to run things on other people’s hardware. Or you need that 7th user. Or you need paywalled features. No worries. Headscale is a self-hosted service that can do nearly everything the cloud portion of Tailscale does, and if you have a place to host it, you can be your own Tailscale server.

For the client side? That’s the best part. Headscale works seamlessly with the existing Tailscale clients. You simply have to point them to your server instead of the defaults.

Continue reading “Linux Fu: Heads Or Tails For VPN”

Linux Fu: Upcycling An Old Router

You’re wandering through a thrift store and spot an old router for ten bucks. Worthless, right? But in this case, it was a Google OnHub, which, at the time, was pretty premium and still isn’t anything to sneeze at. Of course, Google abandoned it long ago, and it runs Chrome, so pass, right? Of course I didn’t. In fact, I bought two for less than $20. The question is always the same: what do you do with it?

OpenWrt will run on the device. That’s a good start, but merely replacing the firmware isn’t much of a project. The more interesting question is whether the hardware can still do something useful. I had a specific need: connect a wired workstation to a reasonably distant WiFi network without running cable and without suffering the usual double-NAT headaches that come from turning the router into yet another subnet. For this, the OnHub turned out to be nearly perfect.

The Hardware

The OnHub was Google’s first Wi-Fi router, built by TP-Link and ASUS in different versions. Mine was the TP-Link model, and one was missing a bit of plastic cowl trim. Under the hood, it has a Qualcomm IPQ8064 dual-core processor — a dual-core ARMv7 — multiple radios, gigabit Ethernet, and enough memory to run OpenWrt comfortably: 1 GB of RAM and 4 GB of flash. The processor also has two network offload processors, but it isn’t clear to me that the stock OpenWrt build uses them.

These devices were expensive when new, but now show up regularly at thrift stores and surplus sales. Installing OpenWrt was straightforward. You do need to remove a screw that covers the magic switch at the bottom, but that’s not a big problem. You can just peel the rubber foot back if you don’t want to remove it. However, the interesting part came afterward.

Continue reading “Linux Fu: Upcycling An Old Router”

Linux Fu: Fake Webcams, GUI Edition

Previously, I looked at using the Linux video loopback system from the command line. The basic trick was simple enough: capture video from a real camera, process it with something like ffmpeg, and write the result to a fake camera device via the v4l2loopback device. Then a browser, or any camera-enabled software, sees the fake camera as if it were real. This allows you to manipulate video before sending it to the rest of the world.

That works, and for those of us who like command lines, it’s easy enough to execute. But not everyone loves the command line. In the comments, there was another obvious answer: use OBS Studio.

While OBS is excellent, it is also a bit like using a laser to chop a carrot. If you already use OBS, fine. If you only want to crop a webcam, add an effect, mirror an image, or feed a virtual camera, it can feel like a lot. If you must have a GUI, you can try Webcamoid, which sits somewhere between a simple webcam viewer and a full video production system.

Webcamoid gives you a GUI for selecting a camera, applying effects, and sending the result to a virtual camera. Conceptually, it is much closer to the command-line loopback setup from the previous post than to OBS. You are still building a pipeline from input camera to output camera, but now you can do much of it with buttons and menus instead of shell commands.

That’s in theory, of course. Implementing Webcamoid turned out to be quite the exercise. Granted, this probably varies depending on where you install software. If your distro has a clean working copy of Webcamoid and its dependencies, good for you. For everyone else, keep reading.

Continue reading “Linux Fu: Fake Webcams, GUI Edition”

Linux Fu: Taming Strace

While many operating systems seem to try to prevent you from peeking under the hood, Unix and Linux positively encourage it. One great tool that we’ve looked at before is strace. Using this tool, you can see details about every system call a program makes. As you might imagine, for any significant program, the output from strace can be huge.

While I’m not always a fan of GUIs, this is one of those cases where making the data easier to browse is a great idea. Enter strace-tui, a text-based GUI for strace from [Rodrigodd]. The program can parse output from strace or manage the strace execution itself, and either way, display the data in a useful way.

I started out looking at [janestreet’s] strace_ui, but the OCaml setup was throwing errors for me, so I just gave up. The strace-tui installs like many Rust programs, using cargo, and it went smoothly.

Continue reading “Linux Fu: Taming Strace”

Linux Fu: Fake Webcams Have Many Uses

Dealing with text streams is a fundamental skill for the Linux power user. You can sort, merge, and search text files easily from the command line. What if you could do the same thing with video? Well, you can. Maybe you want to add a logo to a webcam feed before sending it to a conference app. Maybe you want to blur, color-correct, or annotate video in real time. Or perhaps you want to inject prerecorded video into Zoom while pretending it is a live camera. Linux can do all of this, and the key ingredient is usually the same: a loopback video device.

The basic idea is simple. Instead of an application reading directly from /dev/video0, you create a fake camera device using the v4l2loopback kernel module. Your software pipeline writes processed video into the fake camera, and applications read from it as if it were a normal webcam. The result is surprisingly powerful.

Continue reading “Linux Fu: Fake Webcams Have Many Uses”

Linux Fu: The Bluetooth Regression

There’s a line in a [Weird Al] (no relation) song that says, “I upgrade my system at least twice a day…” I know how that is. I primarily use a rolling distro, OpenSuse Tumbleweed, and if I’m having a problem that I’m too lazy to run down, it is extremely tempting to do an upgrade and see if it just happens to fix the problem.

Of course, the problem is often caused by a previous upgrade. Recently, I’ve been having a lot of trouble with the NVIDIA proprietary drivers, so I updated them yet again. After a huge amount of effort to sort out the video problems, I found that the latest kernel didn’t like my MediaTek Bluetooth adapter, which is built into the motherboard’s WiFi chipset.

This post isn’t about how to fix your Bluetooth problem. You probably don’t have the same setup I do, and even if you do, it will be sorted out in a week or two anyway. But how I temporarily fixed this issue is worth documenting. The details are going to apply to Tumbleweed and this particular adapter, but the general approach should work anywhere with any sort of kernel module problem.

My Own Fault

Part of my problem is my own fault, of course. I have a complex disk setup and do not use the recommended btrfs root file system. That means I can’t do the snapshot thing where I can just undo a bad upgrade. If I did, then sure, I should just roll back and wait for an upstream fix.

I do have “normal” backups, but they are not always totally up to date. Worse, I have found that for things like NVIDIA, the user stuff and the kernel module stuff have to match up. That makes it very hard to roll back a kernel with older modules. The modules themselves live with the kernel, but the user space stuff gets pushed out. Or, if you uninstall things, it uninstalls it for all kernels.

Truthfully, NVIDIA and others like that should keep all the user space stuff in a kernel-specific place, and then symlink it at boot to /usr/bin or wherever. But they don’t. In the end, I didn’t want to go through the trouble of rolling things back and decided to push ahead.

Continue reading “Linux Fu: The Bluetooth Regression”