For a quick file transfer between two computers, SCP is a fine program to use. For more complex, large, or regular backups, however, the go-to tool is rsync. It’s faster, more efficient, and usable in a wider range of circumstances. For all its perks, [Michael Stapelberg] felt that it had one major weakness: it is a tool written in C. [Michael] is philosophically opposed to programs written in C, so he set out to implement rsync from scratch in Go instead.
[Michael]’s path to deciding to tackle this project is a complicated one. His ISP upgraded his internet connection to 25 Gbit/s recently, which means that his custom router was the bottleneck in his network. To solve that problem he migrated his router to a PC with several 25 Gbit/s network cards. To take full advantage of the speed now theoretically available, he began using a tool called gokrazy, which turns applications written in Go into their own appliance. That means that instead of installing a full Linux distribution to handle specific tasks (like a router, for example), the only thing loaded on the computer is essentially the Linux kernel, the Go compiler and libraries, and then the Go application itself.
With a new router with hardware capable of supporting these fast speeds and only running software written in Go, the last step was finally to build rsync to support his tasks on his network. This meant that rsync itself needed to be built from scratch in Go. Once [Michael] completed this final task, he found that his implementation of rsync is actually much faster than the version built in C, thanks to the modernization found in the Go language and the fact that his router isn’t running all of the cruft associated with a standard Linux distribution.
For a software project of this scope, we find [Michael]’s step-by-step process worth taking note of for any problem any of us attempt to tackle. Not only that, refactoring a foundational tool like rsync is an involved task on its own, let alone its creation simply to increase network speeds beyond what most of us would already consider blazingly fast. We’re leaving out a ton of details on this build so we definitely recommend checking out his talk in the video below.
Thanks to [sarinkhan] for the tip!
If you want to ditch Linux to improve speed then that’s fine because there is FreeRTOS. I assure you, it would run even faster.
Probably not true on modern hardware, linux has many, many optimizations for memory caching and asynchronous I/O that are not present in FeeRTOS. Also the TCP/IP stack in Linux is the very best in the industry, much much better than LwiP or any other alternative.
Well… If you are talking about network speeds, i can assure you it’s not really easy to beat TCP/IP stack that is available in Linux. Less code and less resource usage does not neccessarily mean “faster”. RTOS is meant to achieve deterministic latency and meet hard deadline requirements, not to achieve high throughput or be “faster” in general. Also using RTOS of any kind means that you need to carefuly tune task priorities to your specific use case.
You may learn something if you read up on Netflix’s decision to favor the FreeBSD TCP/IP stack in their open connect devices.
A way to get real-time network response in Linux is to use the user-space networking (such as Solarflare). Which means pretty much bypassing all of the Linux tcp/ip stack.
Nothing better to do than recreate the wheel eh? Seems short on imagination.
You seem short on comprehension. There was a well reasoned need identified in the article.
Why read the article when it’s so much easier to criticise a straw man?
There’s no article to read, just a very long video to watch.
You’re not wrong BUT if he had made rsync the /only/ process to run (like he is with his go version) then he likely would have gotten the speed he needed.
You might be surprised. I’d assumed the standard sha*sum binaries were well optimized and the chances of a trivial go program being faster were nil.
However, that trivial go program was significantly faster. It’s been a while, but I’m thinking it was a 1.5x – 2x speedup.
I asked around online but nobody seemed to know why there’d be such a gap.
Rsync / SCP seem to be limited by the performance of ssh. I was extremely disappointed with the speed of them moving files between laptops, and was astounded when airdrop moved them far faster.
You have failed the FreeBSD Fan Club and durable ventilator user group for the last time, Rusty. The glibness droids will be by for you at 6; and they pre-drink.
I had to think, why not eBPF and not hobble the desktop like that, but maybe it’s in the vid? All in favor of getting things done on time of course, dunno how they revisions go.
“Michael] is philosophically opposed to programs written in C, ”
Can’t see any logically argued reasoning here.
I wonder if something similar to the boot on a 1.44MB floppy firewall/router bare bones 486 linux appliance I used to use for dial on demand for the home network would be similarly quick, i. e. a light weight openwrt distro or similar on modern hardware.
Go is a disaster for embedded, last time I tried Go helloworld on OpenWRT, it was 1.6MB. Stripping the binary did not helped much.
Ouch. Truth might hurt fanbois, be careful.
Size of binary isn’t a criterion go claims to be good at; it’s not significant on modern PC’s (or servers) and has little or no impact on execution speed. IIRC most of the space is consumed by elf sections that are only loaded in the event that call stack/introspection info is needed.
For places where size matters, use tinygo or a different language.
Wouldn’t Kotlin or some other modern language make more sense than Go if size doesn’t matter?
“use tinygo or a different language”
TinyGO is only for microcontrollers, not for embedded Linux as far as I can read here:
https://tinygo.org/docs/reference/microcontrollers/
PS: I ran the IOT devroom at Fosdem where one american guy made a show with TinyGO running on uCs https://archive.fosdem.org/2020/schedule/event/iottinygo/
The reason for that is that the standard library is always statically linked and there is zero link time optimization. The anyone who has investigated the Go compiler has always concluded that it’s cutting edge for 1992.
Go requires a garbage collector, so it is a no go for embedded/real-time anyway, regardless of the binary bloat.
25 years from now we’ll still write code in C while those corporate-funded languages will share the fate of Delphi, Ruby or VB.NET
This could be great for a stand-alone backup NAS. Minimal attack surface because minimal software included, and it downloads files to backup using rsync+ssh from the other computers on the network. At least classic rsync has a way to force read-only access, to make it difficult for an attacker to modify or delete your backups.
Systems get more and more hell with more and more TBytes of languages and toolchains needed to survive.
Less is more!
Rewirite it in C!
While it’s an interesting project, it was frankly not a very good talk. About half of the time was spent on the background and motivation, and then just (slightly exaggerating) “I implemented the protocol and here’s how fast it is.”
What is the size of the executable and all dependent libraries compared to the C version? What is the memory footprint when idle and when trasferring files? What is the relative CPU load?
Kudos for using the non-commercial media.ccc.de link first and only using Youtube for the embedded clip. (no sarcasm)
“[Michael] is philosophically opposed to programs written in C” — Well, I am philosophically opposed to people being philosophically opposed to particular programming languages. Less dogma, more consideration.
It’s a little more nuanced. From the blurb on the linked talk:
“I have become philosophically opposed to running C software in my home”
The “in my home” qualifier is actually fairly important here. He is one person, so his workflows need to be fast. Writing projects this large in C would be massively time-consuming for one person, especially if they need to be memory-safe, which networking code really should be (stack/buffer abuse is one of networking’s largest attack surfaces).
In listening to the talk, he doesn’t mind leaning on the C-based kernel, as that is developed by hordes of people, and exploits are quickly addressed.
Go is a great tool, and it fits his purposes handsomely. It definitely isn’t a panacea, and it doesn’t seem he thinks it is either, it’s just the right tool given his skillset and needs. I didn’t see any dogma in his talk or the complete quote, though the trimmed quote in this article does suggest it’s there.
‘faster than the version built in C, thanks to the modernization found in the Go language’
I may be old, but what? How is ‘modern’ a factor in this?
Probably threading and asynchronous IO. The classic rsync doesn’t transfer while it is reading files, though system TCP buffering does allow some simultaneousness.
Last I checked, those capabilities were present in C too, but hey, some people need a pet project, and rewriting from scratch can be more fun than updating someone else’s code.
It’s easier to write event driven code in go using coroutines (less overhead than threads). You can do the same thing in C using select() to determine when one of a set of sockets is ready for reading or writing. You could rewrite the C version of rsync to be just as fast as this Go version. It’s not faster because of Go, it’s faster because of the change in design to avoid blocking I/O, which is easier to do in Go than in C.
Yeah, that’s the crux of it. It’s essentially possible to do anything in C that one could do in other languages (from the processor’s standpoint, not a development statdpoint), and assuming perfect implementation and enough time to implement, it will be faster than almost anything else. The issue is, for applications like this, a perfect implementation won’t really exist, and the time to approach perfection is very high.
“You could rewrite the C version of rsync to be just as fast as this Go version.”
heh that actually crossed my radar. the android supersu app runs su through its own tcp connection…if supersu’s write buffer fills up then it blocks everything until it drains, but rsync doesn’t do a good job of draining under all circumstances. i briefly thought about fixing supersu, or fixing rsync, and finally settled on making my own stupid program to put in the pipe that just always drains both its inputs and never blocks. just a select loop and a couple buffers.
it’s not so hard to design something properly to use select. heck, i don’t think it’s that hard to design something to properly use threads in C or java (i don’t have experience in Go). but these days if there will be any interesting I/O at all then i start there, i think about the async event model before i write anything else. unfortunately from my quick glance at the rsync code it kind of looks like the fact that it would be running over the network was an afterthought to them.
i do know that even programs that use a lot of threads (like a web browser or PDF viewer) often do an extremely poor job of it. i have written a few programs now explicitly around the goal of never ever blocking UI and it is an achievable goal that most user-facing software fails to even approach. it is really dramatic to see software that responds immediately to user input, compared to the status quo today.
but it’s a tricky thing to get right and it seems to be rare that the core developers have the necessary concentration of expertise, authority, and available time. i know i’ve only really developed the comfortable expertise in the past 5-10 years even though i’ve been working with threads since 1997.
not sure what it really says about rsync, or about go, but i think asynchronous programming is a fun thing to think about.
Yes indeed. It is not a language thing (that is a matter of taste on the part of the guy who tackled this, but tastes differ and arguing about them accomplishes little). It is a strategy thing. Select is a pain in the ass, but it certainly is a workable way to avoid blocking IO (and that is why it was invented back in the day). A decent thread library is the better modern solution. And guess what? There have been solid threading libraries for C for quite some time.
That’s the sort of thing I expected. It was the rewriting that improved it, not the change in language.
Why do people still program in dead languages? Program it in java!
It should run in java, runnning on a java interpreter written in java, running on an virtual machine written in java.
More recursion == more speed!
it may take a little while to boot, say around 3 weeks or so, but think of the speed! it will only get faster!
All you need is an 8 core cpu, 32gb of ram, and just a few hundred GB for helper libraries!
It will be super secure, no buffer overruns, etc!
FWIW, the needle on my Sarcasm Detector deflected a little bit off of “Null” while reading your comment.
Ha ha. Java. Is this comedy or tragedy? Plenty of both.
Why is eclipse such a dog — Java!
Well, we needed something to stack up alongside COBOL, so Java was invented.
i’m lukewarm on java when it comes to bloat and performance but nonetheless i have to counter this slander :)
eclipse’s problems stem from a very artless application of OOP patterns when it was first written at IBM. eclipse is its own blunder. java does not have to be that bad
I bet dolla bill gates couldn’t write a line of code. Only code he seems to be wanting to write is to rewrite DNA code.
Strangely, coding is not a key skill for running a large company. Management skills are.
Who’d have thought?!
“[Michael] is philosophically opposed to programs written in C”
Philosophical opposition to a development language is, in my opinion, an odd excuse for rewriting a tool that’s been in use in countless production environments for decades, is stable, and works. But, it’s a cool project and he’s having fun with it, so, more power to him.
“…the only thing loaded on the computer is essentially the Linux kernel, the Go compiler and libraries, and then the Go application itself”
Maybe I’m just old school, but, I always considered having a router/firewall with a compiler installed to be a very bad security practice.
“Maybe I’m just old school, but, I always considered having a router/firewall with a compiler installed to be a very bad security practice.”
+100
You are definitely oldscool, nowadays you also put composer, gradle, npm, nuget et al as well as git onto your production system.
Can I not see the dot or did his isp really upgrade to twenty five gigabits per second? If so where and what isp I need this :)
Chances are it was a typo and in reality it was 2.5 Gbps, which would be much closer to reality.
It really is 25 gigabit. The ISP is Init7.
The biggest issue I have had with rsync was its cpu usage, and as others have said the limitation of ssh. I have found some rsync parameters have helped
https://gist.github.com/KartikTalwar/4393116
I use Allway Sync at home, to back up files. The biggest slowdown comes when I move a bunch of files to a different subdirectory. The program erases the old copy, then rewrites the data in the new location.
Does Rsync (or any other app) recognise data movement / relocation, and speed up the sync process accordingly?
For backup purposes try Borg Backup. It heavily deduplicates chunks over all files so moving or even copying files to other locations virtually take almost no space. Plus it has inbuild encryption support. Won’t give you a readable copy of your files though. You need the borg cli to list, recover or otherwise work with them. https://borgbackup.readthedocs.io/en/stable/