When we first saw [Ben Jojo’s] post about the Internet inside EvE Online, we didn’t think we’d be that interested. We don’t play EvE — a massively multiplayer game. But it turns out, the post is really about understanding BGP (Border Gateway Protocol) and how it helps route traffic in large networks. The best part? He actually simulates a network with 8,000 nodes to test out what he’s talking about.
Obviously, you wouldn’t want to fire up 8,000 Raspberry Pi computers for such an experiment. Using Buildroot, he set up a very small Linux image that had the bare minimum required to run the tests. The qemu provided virtualization, including an obscure feature that allows you to transfer data between virtual machines using UDP. The whole thing ran on some pretty beefy hardware in the cloud. Sure, you could have provisioned 8,000 cloud instances, but that would run into some serious money pretty fast, we imagine. As a wrap-up, he even uses BGP to model his local mass transit system.
While it might seem odd to model a transit system using BGP, this isn’t the strangest thing he’s done with it. If you want to follow along though, all the code is sitting on GitHub.
If you haven’t dealt with large networks before, you might not be familiar with BGP. Most people understand how DNS converts host names to IP addresses. But how does a packet addressed to an IP address in New York City find its way from your computer in Pasadena to the Big Apple? You can find out in the post, but the simple answer is that BGP builds routing tables so that a gateway you connect to can look up the IP address and discover a path that starts with a router it connects to and ends with one that connects to the New York server.
It turns out that even if you don’t play EvE, the post is a fun read and you’ll learn something about how traffic flows on the Internet that you could apply to big systems you might create. Even if they are virtual.
You could also use net namespaces to get a new tcp/ip stack each and interconnect them by using openvswitch. I’ve no tried to get 8000 of them though…
The least expensive way to simulate something of this size would probably be ultralight containers using nspawn. I haven’t been to 8k containers before, but been past 1k on 32 cores and 64gb of memory with pretty minimal resource impact. If there is heavy IO generated this won’t work though, as this is the only real bottleneck.
In any case you don’t need the cloud, you can do this on pedestrian server hardware.
If you have unused server hardware at hand, you can. For such experiments, cloud is the best. If I had unused server, I’d quickly find use for it.