As far as consumer network hardware goes, we’re all expected to be pretty happy with 802.11n WiFi and Gigabit Ethernet over Cat 6 cables. For most home users, that’s plenty of bandwidth for streaming movies and posting K-pop fancams to Twitter on a daily basis. If you want a fatter pipe, things can get expensive, fast. However, [TobleMiner] found a way to use surplus server-grade cards in a regular PC – providing huge bandwidth on a budget.
HPE’s FlexibleLOM standard consists of a special edge connector on HPE servers that lets the end-user fit a variety of network adapters in a form factor designed specifically for blade and rack mount servers. At the electrical level, it’s simply PCI-Express 8x. FlexibleLOM network cards are built for high-speed data center use, often featuring SFP+ and QSFP+ interfaces capable of 10 gigabit and 40 gigabit speeds, respectively.
These cards can be had for under $20 on eBay, but won’t fit in a standard PCI-Express slot. Enter [ToberMiner]’s adapter, which hooks up the relevant PCI-Express lines to where they need to go, and mechanically adapts the FlexibleLOM hardware to fit in a regular ATX PC case.
It’s a great way to get server-grade network adapters in your home rig, without breaking the bank. We’ve featured other attempts at high-speed home networking before, too. If you’ve got the low down on a great way to get multi-gigabit speeds out of cheap surplus hardware, you know who to call.
[Thanks to Marco for the tip!]
Gosh, I wish I had Cat 6 in this home. I’m not even sure it’s even Cat 5e here, but even if they ran Cat 5 for the phone jacks I at least am getting gigabit off of one run.
I could see using this in a rack or a dedicated run between floors if you can easily run a wire or two in a straight shot down a wall, but I think the biggest hiccup for most folks is going to be the cabling. Not to mention 10Gb/40Gb adapters are going to be expensive if you’re not just doing SFP+ cables between adjacent machines.
Rather interesting that the HP connector doesn’t include detect pins like PCIe does so you have to have the 4x/8x selector on it. Kind of janky but whatever you gotta do I guess. One of those things where you really see what a company considers important by what they leave out from a standard and what they add. And I didn’t know that was a good way to do PCIe vias but it makes sense, as it’s maintaining symmetry. Still, pretty neat, if you need the speed!
This looks too good to be true but you have crossed a great bridge sir! I was able to find the card based on the 682148-B21 but will any RJ45 transceiver work in that card? Are there Windows 7 and 10 drivers for the HP card?
The card uses a normal Mellanox chipset. Thus the Mellanox driver can be used once the card is crossflashed. Flashing is required since by default the card is configured for Inifiniband only. The tools required for flashing are available, beacause Mellanox used to publish them until a few firmware revisions ago.
i’m wondering if its worth the effort since USB4 is coming at 40Gbits soon ?
It would be so cool if USB to fiber optic became a thing, and we started seeing consumer grade fiber that was rugged enough for everyday use.
Of course I have literally no use for that much speed and not much use for long distances that couldn’t be done with directional 5GHz, and there’s no POE on fiber, but it’s cool nonetheless.
Fiber cables really REALLY hates being kinked.
That’s the main thing prohibiting it from being a ordinary consumer thing, because people are careless idiots who treat new tech like old tech and then complain when they broke it.
Well… I did some tests while practicing with scrap fibers. It’s not so easy to damage ’em, and a Cat6 cable treated the same way would most probably get damaged, too.
Really depends on the quality of the fibre – a good optical spdif cable can take much abuse and keep working, perhaps even as much as a good copper cable (though I’d say not in my experience its not that far off). The standard easy to get optical ones though are really garbage – stupidly stupidly delicate and still damn expensive (but if you are careful enough with them they work great – just don’t move them!). Where a standard copper cable will be dirt cheap and take heaps of abuse despite that.
I’ve been through countless optical cables for audio, having moved the whole system or replaced kit a few times (some have survived but many have not at least 50% failure over my use of them – but the ones that cost near as much as the damn amp are mostly doing well).. The cheapest and the most high quality of my copper audio cables are all still going strong unless they have a really good reason to fail (like something heavy and sharp edged (monitor arm) falling on them against a hard surface (the desk) when moving things around (the screw clamp had sheared so once the monitor was off the arm there was no weight wedging the arm tight against the desk).
> good optical spdif cable can take much abuse and keep working
Isn’t SPDIF optical cable just a long piece of plastic?
If you want point to point host connectivity, then maybe USB works fine. But if you want a switched fabric, what do you do?
Excellent work. I have one of these adapters lying around.
My home wiring is definitely not the slowest part of the link. My internet connection can just barely handle the amount of K-pop I stream… I mean manly things… Yes, manly things, not that…
Yes, there will be a definite concern that the ‘volume’ of your home will out traffic the IP service.
For most of us I think this kind of project is more useful for speeding up the connections between our devices rather than getting a faster connection to the outside world.
The issue is that these cards are SFP and therefore you need the SFP modules which are much more expensive than the cards as confirmed in the twitter thread which the article links to.
True, the SFPs are more expensive than the card. However, 10G BASE-T equipment is really expensive, when compared to SFP+ hardware. And when you want to go 40Gb/s you have no other option than QSFP+. Also SR SFP+ 10Gb/s transceivers are just ~$15 and duplex fiber is rediculously cheap < 50ct/m. Thus going with SFP+ is usually still cheaper than using 10G BASE-T.
Power consumption of SFP+ cards is usually also quite a bit lower, since they need less signal conditioning and post/pre-processing
10Gb SFP+ adapters are pretty cheap at fs.com. No affiliation, just a fan of inexpensive optics.
Can confirm. My work uses transceivers from FS (primarily with Cisco equipment) and they’ve been very reliable. Also a fan of inexpensive optics.
Depending on what you’re doing, you can get SFP-to-SFP copper cables for short runs that aren’t expensive.
SFP+ is dirt cheap.
https://www.fs.com/c/10g-sfp-plus-63
SFP+’s are not so expensive (on Ebay). However, SFP+ cross-compatibility is a minefield. It’s a very good idea to check known-compatibility data before buying anything unless you want to have lots of useless parts sitting around.
I’ve used surplus server cards now and then but their performance and age generally comes at a steep power penalty. One server essentially doubled its power consumption by adding a four port NIC. It can be an interesting proposition but do your homework to avoid surprises.
And that is exactly why those cards are so neat. They are Mellanox ConnectX-3 which have been released in 2012. That makes them relatively recent. The SFP+ card shown in the photos above does not need any additional airflow over natural convection to operate below 50°C despite its small heatsink. This makes the card lower power than most 10G BASE-T consumer cards.
2012 is not recent in terms of power consumption. Chips from that era are woefully inefficient compared to more modern ones.
You realize that both ends and the switch between them have to also be 10GB to actually communicate at that speed, right?
a common usecase is to do direct client-to-client, e.g. your workstation to a NAS. that’s what I do; 10Gb SFP+ fiber in my desktop, and in my NAS, for iSCSI. I only have a 100Mb/s connection to the internet anyway…
I have a 20Gb/s LACP between the network switch and my Synology Rackststion and a 10Gb/s link from my desktop to the switch. The rackstation has all the bandwidth it needs for cameras, file sharing, Plex, music and running a Roon core. The desktop can shift lots of files to the NAS for backup much faster than it could on 1Gb/s CAT6. The fibre cable from the house to my garden office was the most expensive part of the whole setup at nearly £400. All the other stuff – switch, server adapters, SFP+ modules were bought used on ebay for a very modest outlay. I don’t think it will be too long now before fibre starts to find its way into the consumer market.
Another great and inexpensive source of 10gbit networking are Mellanox ConnectX-3 cards from ebay combined with a direct-attach SFP+ copper-cable and a small SFP+ switch, f.ex Mikrotik CSS326-24G-2S+RM switch with 2x SFP+ ports and 24x RJ45 gigabit for as low as ~100$ depending on where you are in the world.
My workstation and FreeNAS-server is connected to the 2 SFP+ ports, giving the NAS the ability to communicate with up to 10 hosts at full gigabit speeds, and my workstation at 10gbit. Freenas-machine is also setup as router, so it can handle a shitton of connections and small packages without throttling like the ISP-provided routers, and a lan-cache, so windows updates, steam, and a lot of other bandwidth-heavy applications are cached and only consumes internet the first time it is downloaded + the second time, it downloads at link-speed, with up to 10 clients at a time. I mainly use the cache as quick-loading games, and then only having a few installed on my NVMe-SSD at a time. I know, i could just install them directly on the NAS and move them, but this is just as easy, and with this, the wife and when i have friends over for gaming, can also benefit. And >1 gigabit speeds from steam always impresses the ladies…. or at least young gamers…
I’m out of the loop here. What’s the advantage of using this type of card over one of the standard PCIe 10GbE cards that are also on ebay for pretty decent prices?
Have you ever tried 331FLR(BCM5719) on this card? 331FLR can boot into webbios but will see wrong PCIe device in windows,won’t link up but with the LEDs on, even sometime it’ll cause system freeze. I’ve tried 366FLR/544+FLR/561FLR on my own convertion board, they are all working fine. I think the only difference between yours and mine is the #PERST signal. Thank you!
anyone manage to get a 530FLR working? I have a 560FLR working.