[SpookyGhost] has a big home network, and has taken cable management and server organization to the extreme. He has written about individual components before, but this blog post brings it all together and reviews the entire system. The networking gear is installed in a closet and mounted in a 25U tall 19-inch rack. From top to bottom, here is a brief list of the gear:
- Keystone patch panels
- pfSense Firewall / Router
- Two Cisco Ethernet switches
- Redundant internet connections
- Shelf of numerous servers
- RAID-Z2, 12 each 8 TB SCSI, media storage
- NAS RAID, 6 ea 4 TB SAS, 2 ea 800 GB SSD
- Video Management System, 48 TB storage
- UPS and power distribution units
Most of the Ethernet uses 10GBASE-T and Cat6 cabling and connectors, with some interconnects use fiber optical cable and LC connectors. Unsurprisingly, as this setup grew and grew, [spooky] had to pipe in air-conditioning to the closet.
This is a serious installation, but there are plenty of good ideas for folks with less ambitious networking goals and/or requirements. We liked the swappable Keystone jacks in the patch panels, and the cable pass-through panel with a dense curtain of rubber fringe to keep things looking tidy. If you have any ideas to share on network equipment and cable management, let us know in the comments.
Impressive electricity bill that must be for a home server/network.
I guess he lives in Texas. That would be around 0.09 USD per kWh.
Now imagine living in Germany with 0.44 USD per kWh.
The only thing you could afford to run 24/7 here is a Raspberry Pi.
That’s only the “transportation fee” in Denmark (the $0.44)…
Where in Denmark do you live??
The tax is 0.13 USD (0.87 DKK).
The transportation fee is only 0.02 USD (0.14 DKK)
The actual electricity price is around 0.15-0.3 USD (1-2 DKK)
If you pay a lot more you should switch electricity company.
Wrong.
The 0,14DKK is only for the backbone transportation – then you have you have to add the distribution fee, which in the Zealand region is from 0,19DKK (nighttime) to 1,7DKK (primetime winter).
And it makes no difference switching eletricity company, as there is monopoly on distribution.
you should imediatly look for another supplier! (eg. browse through verivox or check24)
the cap of 0.40€ per kWh is just a scam most of them just got complacent on. new contracts can be had (different per region ofc) down to 23ct per kWh. and the switch itself is done in less than half an hour.
still, i also would only run a mini-server, down most of the time till woke on Lan. just because you can afford sth. doesn’t mean you should ;)
If he’s in Texas, it’s the 9 cent/kWh… until you have a cold day when the electrons will run you a jaw-dropping $5/kWh! When the Texas power grid fell flat on its face like a speedwalking drunk, it exposed people to spot-market rates that shot up like a jet plane on steroids – if they were lucky to have electricity.
Texas e-seceded to avoid interstate regulations so they have their own isolated grid waiting for a chance to do a repeat performance of the ice storm debacle. Note to our (least) favourite White South African American: Texas is the WORST place in the USA to build a server farm! Not to mention it’s yet another reason to never visit Texas except for an emergency landing (and the pilot had better have a good reason to land it there)…
Install more of those exhaust fan in the room, and heat insulation pipe for the heat sink and the ventilation, and stay away for the risk of fire. About those hardware like someone had mention, don’t let be redundant of those Hardwares, it will add up spaces and thus it will be less worthy.
That’s one of APC’s SmartUPS line that start at 1.3kW. The Locking connectors on the back wall make me think it’s a SMT3000RMT2U which is rated at 2.7kW. So…
Texas ($0.13/kWh): $3,074.76
Washington ($0.13/kWh): $3,074.76
Georgia ($0.14/kWh): $3,311.28
Ohio ($0.15/kWh): $3,547.8
Pennsylvania ($0.18/kWh): $4,257.36
Alaska ($0.21/kWh): $4,966.92
California ($0.32/kWh): $7,568.64
Hawaii ($0.43/kWh): $10,170.36
WA varies due to all the hydro power and how close you live to a dam. Seattle 13 cents. Bridgeport 3.5 cents. Yes, Bridgeport attracted crypto miners for awhile with commercial rates that drop below 3 cents.
Yes, everyone power is going to vary. I specifically chose average prices where I could. I live in So-Cal and our peak rate is $0.46 but the state average is $0.32. In Pittsburgh the average is ~$0.08 but rural areas are much higher raising the average.
It’s still cheap in Bridgeport. Maybe even cheaper than you said. However if you are only taking power and not providing jobs, they will look at you funny.
A) Residential and Commercial services:
Basic charge: $14.21 per month
Energy charge: $0.0233 first 25,000 KWH
$0.0262 25,001 KWH to 49,999 KWH
$0.0267 50,000 KWH and over
Demand charge: $2.19 per kilowatt of demand in excess of 50 kilowatts.
That doesn’t necessarily mean all of that capacity is being used…
CT – $0.11/kWh (municipal owned electric company aka not UI)
The dude has 17kw of solar
At 12 cents per kWh, 1 watt of 24/7/365 home lab consumption works out rather nicely to 1 dollar per year.
My home network’s base consumption for minimum required functionality is about 66W. If I fire up the lab, it can go way higher, but the VMWare server does most of what it needs within another 60-70W, or 30W when idle. Firing up the wireless lab and the Big Switch (HP 3800) is another 150W.
I shut the Plex server down ages ago because operating it costs more just in power than paying for streaming services. Never mind the cost of the hardware.
This is practically already a small busines network..
Some people have just too much money…
Or it’s a person enjoying their hobby.
I have a very similar looking rack in my basement, and if you select the right amount of equipment, the cost to purchase and run isn’t as high as you’d imagine. My setup consists of a couple 6th gen i7 desktop PCs, a pair of Unifi network switches, and an older enterprise 4u SuperMicro server with a pair of e5 xeons. There’s an older ZT systems 1u w/e5’s that’s a cold backup, which I received for free from a friend who didn’t realize how loud a 1u server would be in a home lab. Paid more for one of my UI switches than did the servers, minus the 14 8tb SAS storage drives.
Running consumption is around 400 watts, which isn’t great, but could be worse. The servers average consumption over the last 30 days is 254 watts which was well offset by replacing all of the incandescent lighting in the house with LEDs.
Nice rack.
That would be a perfect name for a store that sells racks in the French city of Nice ! :-P
Merde! A French shop would never be allowed to use such heinous Englishness.
Begs the question: Do frogs call bobbies étagère?
Living off grid with minimum power storage it’s usually very hard for me to go to “wired” peoples homes… laptops on, tv on, all sorts of RGB crap… Nothing is used… just running..
Nerds that run outdated servers just to “have it”….Don’t know about this HW… but mostly things seem to run for running sake…
I went one better and completely did away with electricity and Internet. I just adjust the oil lamp every hour so I can keep reading the Bible into the long night.
You can mock both things..
But if you see the amount of redundant stuff he has running that is available online he and his kind are the problem of today really..
Built most of the things he has… however once complete I move to next project.
This is like the variant of the rich guy with a home theater and an amusement park all to himself.
While it’s cool and probably a lot he uses… it’s outdated almost colonial way of living..
He could share his internet with his neighbour along with tonnes of other things.. and hey.. maybe built a neighbourhood network.. or wait., some kind of internet…
> He could share his internet with his neighbour
And in less than a month he’d enjoy visit from FBI because his IP was used to spread child porn, extremist content or become a part of botnet. People do weird things when others are not looking.
Less that 5% of people live where the FBI has jurisdiction, so that’s probably not a problem. Just to be clear, I’m dead against child porn, so I’m sure the FBI has it’s uses, just not where most people live.
“daveboltman says:
August 14, 2023 at 6:06 am
Less that 5% of people live where the FBI has jurisdiction, so that’s probably not a problem.”
So he should share his internet with foreigners? Or he lives in a country where there’s no law enforcement?
Did you read the blog post? What part of “T-Mobile”, “Verizon”, and “AT&T” suggests that he’s outside FBI jurisdiction? Leaving aside the fact that the FBI takes over foreign websites on a regular.
From the map graphic *in the blog article* he’s in Texas where the FBI has historically, yes, been a problem, not above burning a building full of children to death. And yes, there was a tank involved.
_______________
This reminds me of 1/3 of a Network Room I used to work with and I was the backup to the guy who set it up. So much heat a dedicated chiller had to be installed. This guy may have “issues” that cause him to be “unusually organized” but I think he’s just really good at what he does and I applaud it.
The nonsense about “He could share his internet” reminds me of Episode 68 of The Twilight Zone, https://en.wikipedia.org/wiki/The_Shelter_(The_Twilight_Zone) . From the pictures he obviously lives in an upscale area, not the barrio or the projects or the rez, so his neighbors’ internet is his neighbors’ lookout, not his. This falls under “don’t borrow trouble” and “certainly don’t try to borrow trouble for someone else”.
Correction to the HAD article: he does not “pipe in air-conditioning” he installed an exhaust fan. “Buh, buh, but air-conditioned air comes in!” Not in winter.
I get your point about wasted electricity and all, but many homelabbers do it to learn networking. Not everyone has access to a NOC, and if they do it’s because they are already employed at one. It’s a fair tradeoff of cost and power use to education and experience gained.
Then there are the weirdos like me who have a 26U rack but use things like Raspberry Pis and mini PCs with external HDD enclosures attached instead of actual server hardware, and the rack is just a great way to organize and hide all the wires so the wife doesn’t complain about that corner of the home office being so ugly. All six of my mini servers and workstations draw less power than one “real” server or full fat desktop PC, and while it would never pass muster at a NOC it’s perfectly fine for what I do with it. As for my gaming PC, it serves one purpose – entertainment for a few hours a week – and it’s powered down when not in use.
“the amount of redundant stuff he has running that is available online he and his kind are the problem of today really”
Assuming you are not just trolling.
Yes most things are avilable online.
Until they are not.
Because the business running them decides they no longer wish to continue that service and cancel it leaving you screwed.
Or they change the ToS.
Or they remove a critical feature (for you).
So yeah, some of us dont like everything being “online” in fact I’m just pointing that out to a client who canned all their offline apps for “the cloud” and it’s broken and lost a ton of people’s projects which they would have had if they were offline still, but they forced them to migrate but obsoleting the offline version.
No backup option in the cloud version – frequent problem with cloud because they (all businesses) want to lock you into their service and their rules.
None of these people pushing cloud think of that, they rarely do, which is why this keeps happening all over the place.
All eggs in one basket. hmmmmm
Once upon a time, everything was cloud, only it was called something different. Then people realised that sucked and rolled their own.
Now business and content providers want you to rent things and NEVER own them. And foolish people thing paying in perpetuity for something is a good idea.
Actually, you sir are the problem of today really.
My thinking too. It is why everything I do is ‘local’ . Local network 24×7. Local data server 24×7. Local desktops 24×7. Off site (sneaker net) backup site. Nothing in the cloud. Way it should be IMHO if you value ease of use, no surprises, no fees, etc…. But that’s me :) . I am not going to force my views on others that think otherwise (unlike some I know).
As for electric usage. Who cares. As long as you can ‘pay’ for the service and justify it. Then use it. Some people own jets, huge houses, big boats, four wheelers… All relative. Never understood the attitude that if ‘I’ don’t want it, then think everyone shouldn’t have it. Goes for a lot of issues too of the day.
As for the article, neat and clean setup! I like it. Overkill for my home use. I do like the ‘wired’ network however. I wish my house had it built in the walls instead of this wireless craze. A bit better privacy and reliability.
Pff. Bible? Where i come from, we had to read theZ cereal box, and swap with the neighbours to get something new…
But to be on topic again:
Some people spend thousands on high end audio, this guy on network gear.
The Bible? You youngins and your new age religions. If it’s not written in cuneiform on clay tablets it’s too modern.
Writing? You whippersnappers! In my day we hit logs with sticks while the shaman told us why the sun was angry, and we liked it.
I used to be that kind of nerd. Back when I was 20-25 years of age. Had an old box running IPCop, another old supermicro box running random stuff. Basically for no reason whatsoever, just to have something to screw around with.
Now? Now I’m 33, My home network consists of a standard all-in-one router provided by my ISP with it’s wifi disabled, two ubiquiti APs to cover the flat, a pair of cheapo 5-Port switches and HPE microserver. The latter might aswell be some off-the-shelf NAS, but I don’t want to touch it as long as it’s running fine.
Honestly, I’ve got plenty toys to play with at work, high-density and high-throughput datacenter, half the servers in there are connected with 2x40G nonblocking. I really can’t be bothered to come home and have to deal with more of that. I am basically the proverbial shoemaker, just with networking.
I also cut down on a lot of computer tinkering myself as my age increased and priorities changed. But not to that extent!
I love my remote access and have never had any luck getting a VPN server to run on consumer router junk. The one time that worked for me on a Linksys something or other it died after just a few days. Ran out of writes to the flash was my guess.
A hand-me-down PC from the office with an extra NIC and OPNSense.. that’s the stuff.
I also fear off-the-shelf NAS. What if it dies, the storage format is proprietary and I can never get my data back off the discs? I refuse to dig up my dearly departed relatives to re-make all our old photos. SBCs are so cheap these days and although not what they are intended for it’s not hard to connect a hard drive to one.
Oh… yah.. almost forgot. I put a power meter on the OPNSense box for a couple of months to get a good average reading. I was worried for both cost and environmental reasons about having a full PC running nonstop. It looks like less than $20 worth of electricity per year. I don’t know how to translate that to °C of climate change but can’t be much!
Some places the water goes over the dam if you use it or not.
I understand why it’s a visceral reaction for you! We don’t know what op’s power situation is nor the myriad other variables to judge how “ethical” it is – nor do we know yours – but even at the very worst, keep in mind this home lab is the equivalent to big cars vs shipping and airlines – one is more visible, but the other carries most of the blame.
The sum total of homelabs is nary a drop in the bucket compared to electricity use in data centers, and nothing at all compared to the completely pointless consumption of energy to support cryptocurrency. Many millions of servers sit idle 95% of the time because it’s cheaper for a company to have them available constantly than engineer boot and shutdown controls. Same for why we build data centers in places that require they are cooled year-round.
Even if op lives somewhere that is coal powered at a high rate, they’re paying residential rates which includes all of the fees related to environmental controls. They may well be living somewhere with extremely cheap power – in a solar-heavy state, or parked up next to a hydroelectric plant.
I’m not here to judge you or him. We all live the lives we can. Just some context that may help you the next time one of these posts wanders across your screen :-)
The OP’s setup uses less power than just the 24/7 lights in just one of the multiple rooms in just one of the multiple data centers we use.
Some rooms we can turn the lights out (our own rooms), but the colocation rooms burn that power aaaallll day and night…
yeah “outdated servers just to “have it”” is how this tickled me as well. i know a guy who worked in a pc repair shop 20+ years ago and he would always have this sort of stuff flowing through his fingers. he was so excited to show me a quad pentium pro machine with a passel of 36G SCSI drives…a real “mainframe class computer” for the era…but he was smart enough, he only plugged it in the once. :)
i’m sad thinking about it, how many things are just so awesome but when you get down to it, it’s hard to even figure out what to do with it even for a brief laugh, let alone justify the expense of leaving it on all the time. it happens on the other end of the spectrum too: i could shed a tear for all of the raspberry pis that got hacked on intensely for a couple weeks and then never used again. i mean, how many of these “pi in a box with a keyboard and screen glued to it” projects on here actually get used after the photo for hackaday?
and then there’s all the things doing a job but they’re dramatic overkill for the job, and/or redundant to idle capacity elsewhere? i used to sit at my pc but then it was just part of my livingroom entertainment center and now it’s in the basement beside the hot water heater. the past just keeps getting farther away…
To me this is just another form of nerd porn, just like watching videos of master machinists and prolific DIYers. Just for the fun of it, over the last couple of months, I went through the house, upgrading the Internet wiring to Cat 6 (and changing a few things to 10 Gbs) and redid the rack with all UniFi and Synology equipment.
I’m using an Asus wifi/router running Asus-Wrt Merlin firmware (the internet company allows me to plug directly into the ONU which is nice), Buffalo PoE switch (because it was cheap and not completely consumer crap), a Synology rackmount NAS, and a Raspi for pi-hole. Plus another Asus as an AP for downstairs.
Cat6A cable throughout the house and it all works quite well.
I thought of using an old Dell server I scavenged from work for something, but the dual old hot and power hungry Xeon CPU’s and the jet-engine fans had it powered down and returned to the scrap bin quick smart.
I still plan to add a UPS, but the power here in Japan is very reliable.
I think it has gone out 2 or maybe 3 times in 7 years, not counting when the breaker tripped due to my own doings…. :D
(I’m more of a risk to my equipment than the power company….. lol )
“HOME NETWORK ORGANIZATION GETS OUT OF HAND”
Out of hand?
It looks very well organized to me!
B^)
This reminds me of my Ethernet switch… The installation guys knew I will setup my network myself, so they just left it running with default settings. I was about to connect to it and go through the configuration, but somehow twelve years have passed.
All connected to 600kb DSL
The next challenge is to improve the energy efficiency… Firewall appliance (<12W) vs. repurposed SuperMicro server, all that NAS equipment… and the air conditioning! In a "closet" scenario, air exchange with the rest of the house should be sufficient. Its just about the amount of air exchange required to keep the temperature acceptable. In the simplest scenario, a 150 CFM 1.0 Sonne bathroom fan. Alas, the cost of doing the above rather than using "repurposed" and surplus parts cost more than any energy savings. In the end, it was done for educational and "fun" reasons.
Forgot to mention the main 48-port Dell X1052p switch
Cool, my server is some hard drives in a usb hard drive enclosure connected to an old laptop.
“Unsurprisingly, as this setup grew and grew”. I was surprised! Unless he became the neighborhood ISP and the customer base is growing.
I have Ethernet to every room and WiFi and hubs and dumb switches here and there. And an HP G9 server on the garage/office wall with one of those vertical hangers, which is a great way to mount one. The server is a 28 core/56 thread with 256G of RAM and SSD RAID. It is used for business and work-from-home people and enthusiasts to build Linux kernels and as a Git host and web site sand boxes, and VM’s and all that there kinda stuff. Also a little NAS with a big drive in an out-building.
I’m guessing all the video sources take all that processing and data flow if you are keeping higher resolution material. I read through the blog and maybe missed something, but I would really appreciate some info on the amount of data being handled and saved, sourced continuously.
I’m having rack envy.
What a great rack….but I guess we are supposed to ignore all those cables on the floor…..
PFSense?!?! You ruinz it!
https://docs.opnsense.org/history/thefork.html
A very tidy rack, really. Good work!
But now not soo extraordinary as some comments make it appear.
A decent house with multiple LAN ports in every room, NAS and things – and the rack is full. It’s not so different for me. Except we don’t have any problems with the power grids over here, so the UPS isn’t necessary – but is part of the solar system anyway. So energy consumption is not the issue ;-)
Home network disorganization gets out of hand…
I really should pull the thin ethernet coax out of my crawl space…but it’s not hurting anything.
I just looked and I’m getting charged $.017022 per kWh. I live in Florida BTW. I need to look at my bill and see whats killing me. I know the A/C is the main one but something else is putting a hurt on the bill. Might be the dryer come to think of it.
The rack looks nice and all but how long will it stay looking like that? I have a full size rack and was looking fleek but after a year it went to hell. I cleaned it up recently but still not as nice when I first had it done.
That’s either a typo or your power is incredibly cheap. I’m paying about $0.11 per kWh.
I’m lucky enough to live in an area where “Evergy” took over all of the local power providers. Fast forward a couple years, suddenly we need to literally *TRIPLE* the rate of electricity to extort an additional 190 million dollars out of their customer base. As usual, the claim is because of increased operational costs and infrastructure improvement…… no way any of that 190 billion gets tacked onto their 850 million dollar profit margin.