Building Faster Rsync From Scratch In Go

For a quick file transfer between two computers, SCP is a fine program to use. For more complex, large, or regular backups, however, the go-to tool is rsync. It’s faster, more efficient, and usable in a wider range of circumstances. For all its perks, [Michael Stapelberg] felt that it had one major weakness: it is a tool written in C. [Michael] is philosophically opposed to programs written in C, so he set out to implement rsync from scratch in Go instead.

[Michael]’s path to deciding to tackle this project is a complicated one. His ISP upgraded his internet connection to 25 Gbit/s recently, which means that his custom router was the bottleneck in his network. To solve that problem he migrated his router to a PC with several 25 Gbit/s network cards. To take full advantage of the speed now theoretically available, he began using a tool called gokrazy, which turns applications written in Go into their own appliance. That means that instead of installing a full Linux distribution to handle specific tasks (like a router, for example), the only thing loaded on the computer is essentially the Linux kernel, the Go compiler and libraries, and then the Go application itself.

With a new router with hardware capable of supporting these fast speeds and only running software written in Go, the last step was finally to build rsync to support his tasks on his network. This meant that rsync itself needed to be built from scratch in Go. Once [Michael] completed this final task, he found that his implementation of rsync is actually much faster than the version built in C, thanks to the modernization found in the Go language and the fact that his router isn’t running all of the cruft associated with a standard Linux distribution.

For a software project of this scope, we find [Michael]’s step-by-step process worth taking note of for any problem any of us attempt to tackle. Not only that, refactoring a foundational tool like rsync is an involved task on its own, let alone its creation simply to increase network speeds beyond what most of us would already consider blazingly fast. We’re leaving out a ton of details on this build so we definitely recommend checking out his talk in the video below.

Thanks to [sarinkhan] for the tip!

Continue reading “Building Faster Rsync From Scratch In Go”

TinyGo Brings Go To Arduino

Go — a modern programming language with roots at Google — is one of the new generation languages that would like to unseat C (and C++) for what we think of as traditional programming. It is only for PCs, though, right? Not so fast! TinyGo provides a compiler that — in their words — is for small places. How small? They can target code for the Arduino Uno or the BBC micro:bit. It can also produce code for x86 or ARM Linux (both 32- and 64-bit) as well as WebAssembly. They claim that a recent project to add ESP8266 and EPS32 support to LLVM will eventually enable TinyGo to target those platforms, too.

As you would expect, there are some subtle differences between TinyGo and the full-blown version. The compiler handles the entire program at once which is slower but offers more for optimization. Certain optimizations for interface methods are not used in TinyGo, and global variable handling changes to accommodate moving data from flash to RAM efficiently. TinyGo passes parameters in registers.

Continue reading “TinyGo Brings Go To Arduino”

The V Programming Language: Vain Or Virtuous?

If you stay up to date with niche software news, your ears may recently have twitched at the release of a new programming language: V. New hobby-project programming languages are released all the time, you would correctly argue; what makes this one special? The answer is a number of design choices which promote speed and safety: V is tiny and very fast. It’s also in a self-proclaimed alpha state, and though it’s already been used to build some interesting projects, is still at an early stage.

Continue reading “The V Programming Language: Vain Or Virtuous?”

Linux Fu: Pimp Your Pipes

One of the best things about working at the Linux (or similar OS) command line is the use of pipes. In simple terms, a pipe takes the output of one command and sends it to the input of another command. You can do a lot with a pipe, but sometimes it is hard to work out the right order for a set of pipes. A common trick is to attack it incrementally. That is, do one command and get it working with the right options and inputs. Then add another command until that works. Keep adding commands and tweaking until you get the final results.

That’s fine, but [akavel] wanted better and used Go to create “up” — an interactive viewer for pipelines.

Pipe Philosophy

Pipes can do a lot. They match in with the original Unix philosophy of making each tool do one thing really well. Pipe is really good at allowing Linux commands to talk to each other. If you want to learn all about pipes, have a look at the Linux Info project’s guide. They even talk about why MSDOS pipes were not really pipes at all. (One thing that write up doesn’t touch on is the named pipe. Do a “man fifo” if you want to learn more for now and perhaps that will be the subject of a future Linux Fu.)

This program — called up — continuously runs and reruns your pipeline as you make changes to the pipe. This way, every change you make is instantly reflected in the output. Here’s the video, here’s a quick video which shows off the interactive nature of up.

Installing

The GitHub page assumes you know how to install a go program. I tried doing a build but I didn’t have a few dependencies. Turns out the easy way to do it was to run this line:

go get -u github.com/akavel/up

This put the executable in ~/go/bin — which isn’t on my path. You can, of course, copy or link it to some directory that’s on your path or add that directory to your path. You could also set an alias, for example. Or, like I did in the video, just specify it every time.

Perfect?

This seems like a neat simple tool. What could be better? Well, I was a little sad that you can’t use emacs or vi edit keys on the pipeline, at least not as far as I could tell. This is exactly the kind of thing where you want to back up into the middle and change something. You can use the arrow keys, though, so that’s something. I also wished the scrollable window had a search feature like less.

Otherwise, though, there’s not much to dislike about the little tool. If writing a pipeline is like using a C compiler, up makes it more like writing an interactive Basic program.

Raspberry Pi On The Go Powers Car System

Most new cars have GPS, rear cameras, and all the other wonders an on-board system can bring. But what if you have an old car? [Fabrice Aneche] has a 2011 vehicle, and wanted a rearview camera. He started with a touch screen, a Raspberry Pi 3, and a camera. But you know how these projects take on a life of their own. So far, the project has two entries in his blog.

It wasn’t long before he couldn’t resist the urge to add a GPS. But that’s no fun without maps. Plus you need turn-by-turn directions. [Fabrice] did a lot of the user interface using Qt5 and QML. He started out running it with X11 but that was slow. It turns out though that Qt5 can drive the Pi’s video directly without using X11, so that’s what he wound up doing. The code that isn’t in QML — mainly dealing with the GPS location — is written in Go, while the code for MOCS (My Own Car System) is on GitHub.

Continue reading “Raspberry Pi On The Go Powers Car System”

Unlocking Drones With Go

Looking for a first project in a relatively new language that’ll stretch your abilities? [Ron] was, so he hacked a commercially available drone and opened up a lot of its functionality, while writing the client software in Go.

The drone is a DJI Tello, which has some impressive hardware like a 14-core Intel processor and excellent video processing abilities. There’s also a vibrant community and a lot of support, making it the ideal platform for a project like this. It communicates to a base station via WiFi, and using some tools like the Wireshark [Rob] was able to decipher a lot of the communications and create a whole new driver for the drone. While the drone can be controlled in the traditional way, users can also write programs to control the drone as well.

The project is both an impressive feat in reverse engineering an inexpensive drone, and a fun example of programming in the Go language. Because of the fun and excitement of drones, they have become a popular platform on which to hack, from increasing their range to becoming a platform for developing AI.

Up AlphaGoer Five

AlphaGo is the deep learning program that can beat humans at the game Go. You can read Google’s highly technical paper on it, but you’ll have to wade through some very academic language. [Aman Agarwal] has done us a favor. He took the original paper and dissected the important parts of in in plain English. If the title doesn’t make sense to you, you need to read more XKCD.

[Aman] says his treatment will be useful for anyone who doesn’t want to become an expert on neural networks but still wants to understand this important breakthrough. He also thinks people who don’t have English as a first language may find his analysis useful. By the way, the actual Go matches where AlphaGo beat [Sedol] were streamed and you can watch all the replays on YouTube (the first match appears below).

Continue reading “Up AlphaGoer Five”