Building Faster Rsync From Scratch In Go

For a quick file transfer between two computers, SCP is a fine program to use. For more complex, large, or regular backups, however, the go-to tool is rsync. It’s faster, more efficient, and usable in a wider range of circumstances. For all its perks, [Michael Stapelberg] felt that it had one major weakness: it is a tool written in C. [Michael] is philosophically opposed to programs written in C, so he set out to implement rsync from scratch in Go instead.

[Michael]’s path to deciding to tackle this project is a complicated one. His ISP upgraded his internet connection to 25 Gbit/s recently, which means that his custom router was the bottleneck in his network. To solve that problem he migrated his router to a PC with several 25 Gbit/s network cards. To take full advantage of the speed now theoretically available, he began using a tool called gokrazy, which turns applications written in Go into their own appliance. That means that instead of installing a full Linux distribution to handle specific tasks (like a router, for example), the only thing loaded on the computer is essentially the Linux kernel, the Go compiler and libraries, and then the Go application itself.

With a new router with hardware capable of supporting these fast speeds and only running software written in Go, the last step was finally to build rsync to support his tasks on his network. This meant that rsync itself needed to be built from scratch in Go. Once [Michael] completed this final task, he found that his implementation of rsync is actually much faster than the version built in C, thanks to the modernization found in the Go language and the fact that his router isn’t running all of the cruft associated with a standard Linux distribution.

For a software project of this scope, we find [Michael]’s step-by-step process worth taking note of for any problem any of us attempt to tackle. Not only that, refactoring a foundational tool like rsync is an involved task on its own, let alone its creation simply to increase network speeds beyond what most of us would already consider blazingly fast. We’re leaving out a ton of details on this build so we definitely recommend checking out his talk in the video below.

Thanks to [sarinkhan] for the tip!

53 thoughts on “Building Faster Rsync From Scratch In Go

    1. Probably not true on modern hardware, linux has many, many optimizations for memory caching and asynchronous I/O that are not present in FeeRTOS. Also the TCP/IP stack in Linux is the very best in the industry, much much better than LwiP or any other alternative.

    2. Well… If you are talking about network speeds, i can assure you it’s not really easy to beat TCP/IP stack that is available in Linux. Less code and less resource usage does not neccessarily mean “faster”. RTOS is meant to achieve deterministic latency and meet hard deadline requirements, not to achieve high throughput or be “faster” in general. Also using RTOS of any kind means that you need to carefuly tune task priorities to your specific use case.

        1. You might be surprised. I’d assumed the standard sha*sum binaries were well optimized and the chances of a trivial go program being faster were nil.

          However, that trivial go program was significantly faster. It’s been a while, but I’m thinking it was a 1.5x – 2x speedup.

          I asked around online but nobody seemed to know why there’d be such a gap.

          1. Rsync / SCP seem to be limited by the performance of ssh. I was extremely disappointed with the speed of them moving files between laptops, and was astounded when airdrop moved them far faster.

      1. You have failed the FreeBSD Fan Club and durable ventilator user group for the last time, Rusty. The glibness droids will be by for you at 6; and they pre-drink.

        I had to think, why not eBPF and not hobble the desktop like that, but maybe it’s in the vid? All in favor of getting things done on time of course, dunno how they revisions go.

  1. I wonder if something similar to the boot on a 1.44MB floppy firewall/router bare bones 486 linux appliance I used to use for dial on demand for the home network would be similarly quick, i. e. a light weight openwrt distro or similar on modern hardware.

        1. Size of binary isn’t a criterion go claims to be good at; it’s not significant on modern PC’s (or servers) and has little or no impact on execution speed. IIRC most of the space is consumed by elf sections that are only loaded in the event that call stack/introspection info is needed.

          For places where size matters, use tinygo or a different language.

      1. The reason for that is that the standard library is always statically linked and there is zero link time optimization. The anyone who has investigated the Go compiler has always concluded that it’s cutting edge for 1992.

  2. This could be great for a stand-alone backup NAS. Minimal attack surface because minimal software included, and it downloads files to backup using rsync+ssh from the other computers on the network. At least classic rsync has a way to force read-only access, to make it difficult for an attacker to modify or delete your backups.

  3. While it’s an interesting project, it was frankly not a very good talk. About half of the time was spent on the background and motivation, and then just (slightly exaggerating) “I implemented the protocol and here’s how fast it is.”

  4. What is the size of the executable and all dependent libraries compared to the C version? What is the memory footprint when idle and when trasferring files? What is the relative CPU load?

  5. “[Michael] is philosophically opposed to programs written in C” — Well, I am philosophically opposed to people being philosophically opposed to particular programming languages. Less dogma, more consideration.

    1. It’s a little more nuanced. From the blurb on the linked talk:

      “I have become philosophically opposed to running C software in my home”

      The “in my home” qualifier is actually fairly important here. He is one person, so his workflows need to be fast. Writing projects this large in C would be massively time-consuming for one person, especially if they need to be memory-safe, which networking code really should be (stack/buffer abuse is one of networking’s largest attack surfaces).

      In listening to the talk, he doesn’t mind leaning on the C-based kernel, as that is developed by hordes of people, and exploits are quickly addressed.

      Go is a great tool, and it fits his purposes handsomely. It definitely isn’t a panacea, and it doesn’t seem he thinks it is either, it’s just the right tool given his skillset and needs. I didn’t see any dogma in his talk or the complete quote, though the trimmed quote in this article does suggest it’s there.

      1. Last I checked, those capabilities were present in C too, but hey, some people need a pet project, and rewriting from scratch can be more fun than updating someone else’s code.

    1. It’s easier to write event driven code in go using coroutines (less overhead than threads). You can do the same thing in C using select() to determine when one of a set of sockets is ready for reading or writing. You could rewrite the C version of rsync to be just as fast as this Go version. It’s not faster because of Go, it’s faster because of the change in design to avoid blocking I/O, which is easier to do in Go than in C.

      1. Yeah, that’s the crux of it. It’s essentially possible to do anything in C that one could do in other languages (from the processor’s standpoint, not a development statdpoint), and assuming perfect implementation and enough time to implement, it will be faster than almost anything else. The issue is, for applications like this, a perfect implementation won’t really exist, and the time to approach perfection is very high.

      2. “You could rewrite the C version of rsync to be just as fast as this Go version.”

        heh that actually crossed my radar. the android supersu app runs su through its own tcp connection…if supersu’s write buffer fills up then it blocks everything until it drains, but rsync doesn’t do a good job of draining under all circumstances. i briefly thought about fixing supersu, or fixing rsync, and finally settled on making my own stupid program to put in the pipe that just always drains both its inputs and never blocks. just a select loop and a couple buffers.

        it’s not so hard to design something properly to use select. heck, i don’t think it’s that hard to design something to properly use threads in C or java (i don’t have experience in Go). but these days if there will be any interesting I/O at all then i start there, i think about the async event model before i write anything else. unfortunately from my quick glance at the rsync code it kind of looks like the fact that it would be running over the network was an afterthought to them.

        i do know that even programs that use a lot of threads (like a web browser or PDF viewer) often do an extremely poor job of it. i have written a few programs now explicitly around the goal of never ever blocking UI and it is an achievable goal that most user-facing software fails to even approach. it is really dramatic to see software that responds immediately to user input, compared to the status quo today.

        but it’s a tricky thing to get right and it seems to be rare that the core developers have the necessary concentration of expertise, authority, and available time. i know i’ve only really developed the comfortable expertise in the past 5-10 years even though i’ve been working with threads since 1997.

        not sure what it really says about rsync, or about go, but i think asynchronous programming is a fun thing to think about.

      3. Yes indeed. It is not a language thing (that is a matter of taste on the part of the guy who tackled this, but tastes differ and arguing about them accomplishes little). It is a strategy thing. Select is a pain in the ass, but it certainly is a workable way to avoid blocking IO (and that is why it was invented back in the day). A decent thread library is the better modern solution. And guess what? There have been solid threading libraries for C for quite some time.

  6. Why do people still program in dead languages? Program it in java!

    It should run in java, runnning on a java interpreter written in java, running on an virtual machine written in java.

    More recursion == more speed!

    it may take a little while to boot, say around 3 weeks or so, but think of the speed! it will only get faster!

    All you need is an 8 core cpu, 32gb of ram, and just a few hundred GB for helper libraries!

    It will be super secure, no buffer overruns, etc!

    1. Ha ha. Java. Is this comedy or tragedy? Plenty of both.

      Why is eclipse such a dog — Java!

      Well, we needed something to stack up alongside COBOL, so Java was invented.

      1. i’m lukewarm on java when it comes to bloat and performance but nonetheless i have to counter this slander :)

        eclipse’s problems stem from a very artless application of OOP patterns when it was first written at IBM. eclipse is its own blunder. java does not have to be that bad

  7. “[Michael] is philosophically opposed to programs written in C”

    Philosophical opposition to a development language is, in my opinion, an odd excuse for rewriting a tool that’s been in use in countless production environments for decades, is stable, and works. But, it’s a cool project and he’s having fun with it, so, more power to him.

    “…the only thing loaded on the computer is essentially the Linux kernel, the Go compiler and libraries, and then the Go application itself”

    Maybe I’m just old school, but, I always considered having a router/firewall with a compiler installed to be a very bad security practice.

  8. I use Allway Sync at home, to back up files. The biggest slowdown comes when I move a bunch of files to a different subdirectory. The program erases the old copy, then rewrites the data in the new location.

    Does Rsync (or any other app) recognise data movement / relocation, and speed up the sync process accordingly?

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.