For the average home gamer, good old fashioned Ethernet at 100 Mbit/s is only just starting to become a bottleneck as things like 4K video streaming begin to demand more bandwidth. As always, though, there are those who wish to push the limits of what is possible. [Jeff Geerling] is one such operator, who set out to maximise the network throughput on the Raspberry Pi Compute Module 4.
The build began by taking advantage of the PCI-Express 2.0 single lane interface on the new Raspberry Pi Compute Module. Hooked up to an Intel four-port Gigabit Ethernet card, and in combination with the onboard Gigabit-E port, [Jeff] was able to get 3.0 Gbit/s out of the setup without too much fuss. However, he wanted more, and set about finding where he was being held back. It turned out that ksoftirqd, a daemon that handles network packets, can only run on one core on the Raspberry Pi 4, and it was getting maxed out at this data rate. Overclocking the CPU helped, getting the max rate up to 3.4 Gbit/s.
Further analysis showed that the onboard interface was only contributing 200 Mbit/s, with the Intel card maxing out at 3.2 Gbit/s. In the case of the latter, this was due to the limits of the PCI-E interface. In the case of the former, however, [Jeff] knew that more was available. The trick turned out to be recompiling the Linux kernel to allow the internal interface to be able to set to use a higher Maximum Transmission Unit. This allows each network transmission to carry more data without extra CPU load. With the internal interface and the external card all set to an MTU of 9000, the Pi was able to spit out a scorching 4.15 Gbit/second. Details of the hack are available on Github for the curious.
It’s a hack that doesn’t offer a lot to the average user, though [Jeff] states he has some interesting applications in mind. He’s also contemplating what can be achieved with a 10 Gbit card, which we can’t wait to see. If you want to learn more about the Compute Module’s features, including a couple of tips for laying out yor own board, check out our review. Video after the break.
[Thanks to Baldpower for the tip!]
There is daemon called “irqbalance” to magicaly fix your irq-affinity issues. But i guess it will not help in this case as there are hardware problems. (Perhaps it is not a multiqueue network adapter?)
I tried installing irqbalance, but it would not run. I also tried manually pinning interfaces to different cores, and that didn’t seem to work either. You can see some of the attempts in the GitHub thread linked in the story above; I think, in the end, the PCIe interface on the BCM2711 is just a bit too limited to do some of the things I’d *like* to do with it (I’m also testing GPUs and hitting problem after problem…).
Title is wrong. Should be 4 Gbps, not 4 GBps.
i think this a Hackaday thing.. all titles are all caps. Could change it to 4Gbit/s, though.
Title in all caps: distinction between b and B not possible.
If you copy/paste the title, you’ll see that they used the correct unit:
Getting Over 4Gbps Out Of A Compute Module 4
It’s the CSS. You need something like this as a CSS override:
h1, h1>a, h2, h2>a, h1>a:visited {
text-transform: none !important; // don’t make it all-caps…
font-variant: small-caps !important; // …make it small caps!
}
It really does look better as small caps.
If you’re bus-limited, what more are you oping to achieve with the 10G card? 3.2G across a single wire, or something else?
” 100 Mbit/s is only just starting to become a bottleneck” – I don’t know what planet this is on!! – I swapped the home network to Gbit/s 17 years ago (wired the whole house) and for about the last 10 year or so IT has been the bottleneck!! Copying files and doing things over the network (at 1Gb) is much slower than using the local drive.
I’ve been keep an eye on the price for 10Gbit/s for years, as that will finally solve the problem with spinning drives copying large files around, and won’t be to bad for files on NVme drives (though of course slower than them).
100Mbits/s has been unusable – slower even than your internet connection – for many many many years…
The reference was to the average home game and/or movie streamer, so this bandwidth is referring to the actual requirements for a given application. This is obviously much lower than the total bandwidth you’d want, especially locally. Also, and here I may be wrong, I believe it’s referring to actual bandwidth, not the peak bandwidth that you can get with 100 Mbit/sec hardware (where actual bandwidth is lower than 100 Mbit/sec). Still, I agree, the original statement is not very clearly expressed.
100 Mbps unusably slow? Anywhere else I would think a comment like this is a joke.
My home Internet connection might get as high as 50 Mbps, and even then, only when the planets align.
” swapped the home network to Gbit/s 17 years ago (wired the whole house)” Ya sure you did. 17 years ago? I don’t think so. Keep living that dream.
Gigabit Ethernet was released in 1998, so wiring a house with Cat-6 in 2003 is perfectly feasible. But hey, keep trying to win fake internet points by trolling the comment section, I’m sure you’ll get there one day.
Come to Australia mate. Some people here are lucky to get 20Mbit down, 1Mbit up. XD
You get 20Mbit down? Where you living mate? Sussex Street in Sydney? :P I’m lucky to get 8
You are saying your LANs are slower in Australia? I don’t think this is about his internet connection.
Well there are almost always wireless community networks as alternative if your ISP suck so much.
https://en.wikipedia.org/wiki/List_of_wireless_community_networks_by_region#Australia
https://github.com/phillymesh/meshlocals/blob/master/meshlocals.md
100Mb is unusable he says…. Me playing games with my friends online running at 100. My friend in Alaska running at 10 with a data cap. Yep 100 sure is unusable. I don’t even see what you are doing, what are you copying things onto the system as you use them. Some of the largest games I’ve seen are maybe 100G, oh no you have to wait a whole minute to download 100G that is unberable!!!
Installing the game on a NAS and playing it that way maybe?
I’d be interested to know what kind of application Jeff Geerling has in mind. Traffic generation or noise generation, perhaps?
I’m looking forward to the inevitable router kit that this hack makes possible. Raspi compute module + custom board = one bad-ass router for cheap, way better than a standard store-bought router, much more configurable.
But what are you going to run on it? Like anything the software is the crux of the issue as support or development dying would make this just like anything else, and I would go so far as to say that hardware options have never been the problem.
If you just want a more powerful router on ARM, there is mikrotik or Edgerouter on the mediatek SOCs, all the way up to Annapurna Labs chipsets with 10gb support.
If you literally want to bake your own from source, Solidrun is a great whitebox player in this space for ARM, and there are a million options if you go x86…
Without a decent router-os there isn’t really any reason to get this over a commercial router and openwrt, or literally anything and pfsense.
It all boils down to the same linux problem… There are too many options so as soon as someone gets a new idea there are 20 branches and development hell ensues from the brain-drain on the existing solutions.
I think the point is that if there was a standard, performant, router platform it would make developing (and using) projects like openwrt/dd-wrt/tomato/etc much easier. Finding a halfway decent router that has guaranteed support for third party firmware isn’t as easy as it should be, and even when you find it, it’s not going to be half as powerful as the CM4.
How about OpenWRT? Or pfSense/OPNsense if someone makes a decent port for the Pi.
The real game changer will be using Docker or some other virtualization solution to implement multiple routers on a single Pi. That can be really helpful if you want to use multiple VPNs on your home network, with a dedicated network segment or VLAN for each one.
What are you talking about? You don’t have to have a dedicated router OS to have a router. Raspbian is just Debian and it doesn’t take much Google Fu to learn how to make Debian act as a router. Or, use any of the other Linuxes that run on the Pi. They all can do it, Linux is Linux. Or install FreeBSD on it. I’ve heard FreeBSD makes a more efficient router though I suspect FreeBSD on a Pi is a much smaller community thus less support.
Sure, using something that is supported by a ready-made image like DD-WRT or Tomato or whatever is easier but hardly a requirement.
This sound like he’s discovered Jumbo Frames
These young ‘un’s! 100BASE-T is “old fashioned Ethernet”?
No!
Old fashioned Ethernet ran at 10M, in a big yellow cable, with vampire taps.
Get off my lawn, you rotten kids. Otherwise I’ll throw 400GBASE-DR4 QSFP-DD transceivers at you until you feel properly shamed calling 1G Ethernet fast. Pfft!
(But seriously, cool hack, guys! Great article.)
Where is you lawn?
I would not mind if you trow a few bucket fulls of those things at me.
Nuh Uh!
Beware self-proclaimed old networking guys throwing equipment.
If he really is old he might sneak a Lancity modem in there, it could be deadly.
I’d take the chance anyways. I have some old 16 port hubs from a local IRS office I can wear as armor.