If you spend your time plotting evil world domination while stroking your fluffy white cat in your super-villain lair, it’s clear that only the most high-performance in computing is going to help you achieve your dastardly aims. But computers of that scale are expensive, and not even your tame mad scientist can whistle one out of thin air. Never mind though, because if your life lacks a supercomputer, there’s one for sale right now in Wyoming.
The Cheyenne Supercomputer was ranked in the top 20 of global computing power back in 2016, when it was installed to work on atmospheric simulation and earth sciences. There’s a page containing exhaustive specs, but overall we’re talking about a Silicon Graphics ICE XA system with 8,064 processors at 18 cores each for a total of 14,5152 cores, and a not inconsequential 313,344 GB of memory. In terms of software it ran the SuSE Linux Enterprise Server OS, but don’t let that stop you from installing your distro of choice.
It’s now being sold on a government auction site in a decommissioned but able to be reactivated state, and given that it takes up a LOT of space we’re guessing that arranging the trucks to move it will cost more than the computer itself. If you’re interested it’s standing at a shade over $40,000 at the time of writing with its reserve not met, and you have until the 3rd of May to snag it.
It’s clear that the world of supercomputing is a fast-moving one and this computer has been superseded. So whoever buys it won’t be joining the big boys any time soon — even though it remains one heck of a machine by mere mortal standards. We’re curious then who would buy an old supercomputer, if anyone. Would its power consumption for that much computing make it better off as scrap metal, or is there still a place for it somewhere? Ideas? Air them in the comments.
You mean the one used for the Stargate? Sign me up!
My first thought as well.
This would be just the thing for a deep dive into the Mandelbrot set!
Or finally getting to the end of pi
Don’t we already know the ending? Undoubtedly, the last 2 digits have to be 42.
No, isn’t pi a number with infinite decimal values ?
You missed the Hitchhikers Guide reference.
Chuck Norris knows the last two digits of pi.
Chuck Norris can divide by zero.
>However, the system is currently experiencing maintenance limitations due to faulty quick disconnects causing water spray. Given the expense and downtime associated with rectifying this issue in the last six months of operation, it’s deemed more detrimental than the anticipated failure rate of compute nodes. Approximately 1% of nodes experienced failure during this period, primarily attributed to DIMMs with ECC errors, which will remain unrepaired. Additionally, the system will undergo coolant drainage.
Nice, a slightly pre-owner supercomputer.
They should have used Fluoroinert.
Lightly used super computer. No coolant leaks if you don’t disconnect anything. 1% of memory repurposed to provide creative answers to data requests.
No lowball offers, I know what I’ve got!
9.9/10
No lowballers or rack kickers.
mmmm… Wonder if you could use the bad RAM as an RNG?
HAHA
>> “Nice, a slightly pre-owner supercomputer”
Ran when parked.
I think the fact that it consumes 22.7 MEGAwatts of power might be an even more of an issue that the eventual winning bid price and transportation costs.
I think you misread the article. Frontier uses 22.7 MW, whereas Cheyenne uses a mere 1.7 MW
Wow! Only 1.7MW Hmmm…. Now I have to do the math. Let’s see. 1.7MW / 3 (3 phase power) = 567KW per phase. At 567Kw / 208V = 2724 Amps per phase. Hmmmm…. just a bit more than my electric service ca do.
I would assume at those levels, 480v 3p would be used.
More likely to use
13KV “raw power” like the big industries do. You can buy the juice much cheaper at the level,
supply your own transformers, and still save money
Your math is off:
1,700 kW ÷ [sqrt(3) x 0.208 kV] = 4,719 A
This assumes a 100% PF which might be the case, but even at 85% it is only 2,000 kVA and at either of these levels it becomes more practical to use 480V.
2,000 kVA ÷ [sqrt(3) x 0.48 kV] = 2,406 A
Then definitely 277v, it’s never worth breaking the 1000v threshold if you are under 3000amps. And 100-300v converters are super easy to find.
I wonder if it comes with a UPS. Because 1.7MW UPS for 40k is a good deal!
Anyone know how much gold is in the processors?
Here in Europe we have 240v per phase and 400v inbetween 2 phases.
UK domestic supply is typically fused at 100A per household (for a normal single-phase 240V install) so somewhere above 20kW available per house. Get a street to band together and you could run your own supercomputer complex (free central heating, but you’re on your own for AC!).
>> whereas Cheyenne uses a mere 1.7 MW
At the US average of $.16/KWh, that’s about $272/hour.
So… I guess the question is how many bitcoin you can mine in an hour with this thing?
$50 an hour here. The question is how much do you charge researchers and how much Monero (made for CPU mining) can you coin. $0.029 a kWh https://www.google.com/maps/place/Chief+Joseph+Dam,+Bridgeport,+WA+98813/@48.0042518,-119.6698144,3736m/data=!3m1!1e3!4m6!3m5!1s0x549c7c6363d1207b:0x578ff27c97daa8ed!8m2!3d47.9951446!4d-119.6403287!16zL20vMDM3MTB5?entry=ttu
Not that many. Probably a single ASIC ( specialized hardware for mining) could mine more bitcoins. ASIC are up to 100000 times better than CPUs in this regard.
But… does it play doom?
Can it run Crysis?
The one at Area 51.
How about a game of chess….
Or Thermonuclear War….
>Would its power consumption for that much computing make it better off as scrap metal, or is there still a place for it somewhere? Ideas? Air them in the comments.
Feels like these older but still very far from eclipsed by consumer hardware supercomputers still have a great deal of life left in them if you can use them wisely. Probably end up tuned down a touch so their thermal output and peak power draw is less extreme and then co-habitated in places that actually need the heat – can even regulate the compute speed to the heating requirements. As why just heat when you can get the heat and another useful product at the same time. There is always the need for some serious but no longer really top shelf computing power, and so many industries that always want definitive warm but oven hot.
With the current growth of AI type stuff looking unstoppable a giant pile of RAM tied even to very slow compute in modern terms is going to be valuable on those giant datasets.
313,344 GB of ECC memory would make one hell of a RAM drive (for any VFX studio). And at $0.127 per GB it is dirt cheap. As for the CPU’s, and other bits and bobs with gold in them you might break even on the shipping and time invested
The electric bill will suck though, someone above calculated over $270 per hour of electricity at 16 cents per kwh. So unless you have a personal power plant or have very deep pocket, this computer would make for very expensive RAM drive.
To mine bitcoin with this thing, you will have to get a shovel and mine uranium, to make the electricity.
Bitcoin hashing fixes this.
Could have done it with a 555 timer.
I think it’s not enough with one…
With 555 pieces of 555 timer then?
556 then, do I have to think of *everything* round here?
Export controlled?? A lot of researchers around the world would be happy to have them!!!
The US government would like to have a word with you about ITAR.
Don’t worry they only want to talk. This time.
Dang!
I wish I had heard of this sooner,!
Two days is not enough time to put a team together and submit a bid.
But…
How does one get 14000 cores by multiplying 8064 times 18?
Oh, wait, they may have used a European system of delineating large numbers with commas, or radix points!
part it out. Thats likely what whoever buys it will do. Individual servers can be sold for decent money on ebay.
I’ll take two, please bag’em and carry them to my Pinto.
Or, based by my early comment, i’ll take 555 of them.
My biggest question is how to bring this to the wifey.
“Dear, you know I always wanted to buy a Porsche as I passed 50, but I got something bigger. We need a larger garage.”
I can easily see those words on the tombstone instead of the classic RIP. And all the servers/racks dumped on that grave.
I already asked my wife, she said yes but only after we win the lottery. Good to know she would let me if we didn’t have the means!
“The system used the SuSE Linux operating system and the Altair PBS Pro scheduler. Accompanying the compute hardware, Cheyenne also …”
I guess you would need a healthy 3-phase supply and significant AC laying on.
440V at you-can’t-afford-it current.
Someone computed $272 per hour with US average of 16 cents per kwh
If anyone wants something a bit more garage-sized, I have a Convex Exemplar laying around! (And for that matter if anyone knows a museum that wants one….)
Are you actually serious? I mean depending on price I guess but I don’t have supercomputer money 😞
I mean it’s a “supercomputer” in name but being from the mid-90s it’s value is basically historical as it’s compute power and efficiency are thoroughly obsolete. I saved it from being scrapped probably 10 years ago, at that time it was already obsolete (but still ran at least). So, the price is “free to a good home” lol (ie I’d like to give it to a museum or collector, not have it melted down for scrap)
Idk which side of the country you live on, but there’s always the CHM in California! https://computerhistory.org/
I may be interested.
What is the single thread performance of this thing?
:)
In my experience the number of threads does not matter much. It’s nice to have a handful of cores / thread, so the OS runs on another thread then your application, but in general there are far to few applications that can even make effective use of the 6 core 12 thread Ryzen 5600G that I have now.
Something like matmul can be spread across multiple cores; if the matrices are big enough, you would see a substantial speed-up.
They are good Xeons, E5-2697. If they are worth $18 each, or $1 a core, you have $142,000 just for the processor chips. A remotely accessed supercomputer that was used for atmospheric simulation at NCAR? Put it in a warehouse in Eastern Washington where you can find juice for 2.9 cents a kWh.
1.7MW just for the computer will most likely exceed any reasonable warehouse power substation…you need a decently sized factory…
There are spacial arrangements for crypto miners near the dams. https://www.google.com/maps/place/Chief+Joseph+Dam,+Bridgeport,+WA+98813/@48.0042518,-119.6698144,3736m/data=!3m1!1e3!4m6!3m5!1s0x549c7c6363d1207b:0x578ff27c97daa8ed!8m2!3d47.9951446!4d-119.6403287!16zL20vMDM3MTB5?entry=ttu
1.7MW is $50 an hour here https://www.google.com/maps/place/Chief+Joseph+Dam,+Bridgeport,+WA+98813/@48.0042518,-119.6698144,3736m/data=!3m1!1e3!4m6!3m5!1s0x549c7c6363d1207b:0x578ff27c97daa8ed!8m2!3d47.9951446!4d-119.6403287!16zL20vMDM3MTB5?entry=ttu
The computer I am currently typing this on runs a Xeon E5-2667 that I paid $15 for. I was thinking about this too, selling the processors and memory would fetch a lot of money.
Yes, I have a used HP server with a pair of 2680 for 28 cores. It is great and cost $500 with rails and maxed RAM. Runs Debian Server like a dream. Any of the SGI blades in this Cheyenne unit would be very nice.
They arent worth $18. They are being listed and slowly selling at $18 in singular quantities. Bulk selling old items with miniscule demand has its quirks, you either sink cash for years and sell slowly hoping nobody else floods market with same item, or you offload it all at once for scrap value.
Wow! I could buy that! But what would I do with it? Hmmmm…. I’m not a mad scientist, and I don’t have an evil lair. I don’t run Grand Theft Auto. I’m not certain what I would do with it. Hmmmm…. I guess I’m going to have to pass on this.
Sure…nor a supervillian, that is exactly what a supervillian would say. Yes. I am also not a supervillian
I am a mad scientist with an evil cat, the cat will love the warm hardware and I can create an Evil LLM 👹
It does not make sense.
According to my math you can match its perf with about 50 RTX4090 class GPUs, or less of latest NVIDIA data center number crunching appliances with much less space and power use. It could be cool to have one in your garage but the proprietary cooling system on it is already disintegrating and memory is developing lots of errors from overheating. So good luck maintaining it. It makes a nice piece of furniture though but it is kind of large.
I’d rather get one Origin 3000 rack, at least I could run some fun old Irix on it.
4090 cannot scale it only has PCIe and not NVLink. Also that supercomputer had ~5 PFLOPS of FP64 precision where a 4090 has ~1TFLOP so you will theoretically need 5000 of these GPUs. But GPUs cannot run everything, you need cps with normal cores, not CUDA ones.
That is why NVIDIA datacenter boxes are better solution for FP64, lets say HGX B200 is 320TF FP64, so you only need 16 of these boxes. Or 10 HGX H200 8 gpu boxes. That will probably fit in 2-3 rack cabinets and use much less power than CHEYENNE, And has all the right interlinks too. And as for workloads, sure, GPUs cannot run everything, but the shit that you run on these machines is not millions of single core threads (which operate on small vectors anyway), it is basically just huge matrix multiply workloads. ;-)
You cannot even sell this shit for parts because no one needs old Xeons or faulty RAM. So gold recovery and scrap value?
Thinking Machines Corporation had the right idea when they did the industrial design for their matte-black, curved-face DataVault RAID. It was plain for all to see that, once decommissioned, it would make an awesome bar counter. And indeed, the MIT Media Lab gave it exactly this “second career” for many years after they retired their Connection Machine from supercomputing (e.g. realtime, holographic videos).
Finally lag free Minecraft hosting. :D
More like hosting a larger number of laggy servers than ever before.
All things considered — 8 years old, inadequately cooled, yet still only a 1% node failure rate? I wish my HPC farms ran that well.
Somewhere around here I have a picture of me doing a me doing a pinup pose (don’t get excited, I have the physique of an engineer) with a rack of servers that looks a lot like the above.
When I built out a supercomputer in one of the labs at Intel, it converted me from a hardware is everything to software and systems analysis. with our burnin phase on the computer array, obtaining code that could reasonably push systems was very difficult. We wound up calculating primes, at the time crypto was not on the radar (It’s been a while). I came away from that gig with 2 persistant thoughts that I have used for guidance… 1. I don’t mind hammering servers, If I am running one at 80% – 100% resource utilization, I don’t mind unless my users are waiting. 2. Most of the glorious machines that we build are massively under-utilized, It takes great coding to really push big iron, and there is not a lot of really great code out there.
I recommend the thought experiment of figuring out what you would do with a computer that big. Even if your power budget is more in line with clusterpi. When I faced one down and was staring at the console of a ready to run machine, it occurred to me that all my problems are small machine problems. Even a massive cluster like that, I expect that between storage, memory, networking, and compute you will bottleneck on one of those issues before you hit 20% on another. So the build out of a machine that big tends to be fairly custom.
Optimizing the massive parralelism in Crypto, LLM, Radio astronomy, protien folding, weather, and some other large problems, makes it so that by the time you know how to do it on a supercomputer, you probably no longer need it, because the latent idle computers can likely do it, and you don’t need a supercomputer for that.
I’m happy to be wrong, I recognize you may have succeeded where I did not, or have solved where I got stumped. I yield to experience and success.
Under-utilization of singular servers is one of the reasons why virtualization and containers are commonplace today. Most singular jobs do not require multiple systems to run, unless we’re talking about siloing specific tasks to specific systems (SANs, compute nodes, front-end/back-end topology for webservices) or fault tolerance. Multiple physical servers may still be required to host the quantity of different jobs within an organization, however, even if multiple jobs can be run on a single physical server.
That said, for those jobs where parallelism is beneficial, you have to consider the overhead for each individual node. Each node adds more complexity to communication which requires more resources to handle. As you scale out, you’ll need more and faster communication to keep a parallelized task up to the speed that a single node could process on its own. The slower a parallel system becomes the more argument can be made that a smaller system (less nodes, or single node) should be used instead.
Cryptomining benefits from parallelization, but generally each individual node (or group of nodes) will handle a sized workload. They don’t have to wait on the response of other worker nodes to continue; they update the pool handlers who then distribute work as resources become available. This is one type of job where work is done across multiple systems but does not require in-order execution to complete. These kinds of pools can be easily decentralized, however there’s an argument that having low latency bandwidth within the pool is still beneficial as it helps decrease the solution time which increases the chance of winning the race to solve a hash. Supercomputers might win more often for the hashes they compete on, but there’s enough work that decentralized pools may have more wins overall.
Large datasets like LLMs may require massively parallelized systems to handle some datasets. This occurs when tasks are hard to atomize (to split up so they only need some of the working memory instead of all of it). Many calculations may need to be done in order (the completion of the previous step is required to calculate the next step), and the working memory may need to be accessible to the cluster as a whole to complete the next step. This would only be required for very large datasets nowadays, and as you put it you may know enough about the problem that you can figure out how to split up the job so multiple smaller clusters can handle the job more efficiently.
So why have supercomputers in todays age? A supercomputer can act as a single resource pool: It can handle job scheduling and systems monitoring across the cluster. Logically, it’s a single system that runs off of hotswappable equipment. As mentioned up above, it has lower latency and higher bandwidth available for internal communications, which may not be available for decentralized systems. You aren’t at the mercy of quite as many external points of failure (read: ISPs), and as each node is both powerful and failure resistant (ECC RAM, server-grade hardware) your system will have less complexity and points of failure to deal with overall.
If you’ve got the work and a lot of it, there’s still a reason for big metal.
22.7 MWh @ 15.45 cents per kWh
That would be $3,507.15 USD per hour, just to run that thing
Around $30 million dollars a year, just on energy consumption
The auction just cleared $100000, and the reserve’s been met. Someone’s going to end up with this.
Guess I won’t bid tree-fiddy then
Bet it still can’t emulate playstaion 2 games well
Finally a machine that can cope with my addiction to not closing tabs in Chrome.
… or at least it should keep me going for a while before it crashes.
I want those front panels for decoration on my basement wall, but not for $100k (at time of posting)…
We sure could use a capable build server where I work…
Selling for parts because I don’t have a power cord to test.
Has a US supercomputer ever been sold and actually reused elsewhere in the recent history of MASSIVE supercomputers. In the cases of sold government surplus, the item must be removed from its location within a specified amount of time at the buyer’s expense. To do that with the care required when intending reuse would be phenomenally expensive. For scrap, just the 8,064 Intel Xeon E5-2697v4 socketed CPUS at $40 each would equal $322k, but that’s at current eBay prices and that many of those CPUs would flood the market, I suspect.