For those of us who like to crawl over complex systems, spending hours or even days getting hardware and software to work in concert, working at places like NASA or CERN seems like a dream job. Imagine having the opportunity to turn a wrench on the Space Shuttle or the Large Hadron Collider (LHC) — not only do you get to spend some quality time with some of the most advanced machines ever produced, you can be secure in the knowledge that your work will further humanity’s scientific understanding of the universe around us.
Or at least, that’s what we assume it must feel like as outsiders. But what about somebody who’s actually lived it? What does an actual employee, somebody who’s had to wake up in the middle of the night because some obscure system has gone haywire and stalled a machine that cost taxpayers $4.75 billion to build, think about working at the European Organization for Nuclear Research?
Luckily for us, Daniel Valuch stopped by the Hack Chat this week to answer exactly that question. In sharing everything from his grueling nine hour entrance interview to the early days of the LHC where failures occurred every few minutes, he didn’t sugarcoat the situation. That said, after spending two hours sharing his fascinating stories with members of the community, the answer seems pretty clear — Daniel loves his job just as much as we thought he would.
Early on in the chat, he says that joining the LHC team as it was still being built reminded him of movies and documentaries he’d seen about NASA’s Apollo Moon program. There was a palpable feeling that what they were working on was of vast scientific and historic importance, and he couldn’t help but be in awe of the brilliant scientists and engineers that he was rubbing elbows with. He might have been the new guy at the lab, but thanks to his RF experience, Daniel soon found himself working with the high-speed ADCs, DACs, and FPGAs responsible for controlling the beam of accelerated particles.
For the home hacker, it might not be immediately obvious how RF skills apply to particle accelerators. But of course, this is RF on a scale that few of us can imagine. As Daniel explains, accelerating and stabilizing the beam requires multiple megawatt-class amplifiers, and a good working knowledge of RF theory is a must. In his own words, “there is no physics without RF and also no RF without physics.”
Even with all the equipment and knowledge in the world, working on a cutting-edge particle accelerator proves uniquely challenging. When dealing with events that are occurring at effectively the speed of light, nothing is really straightforward. Even the speediest of electronic systems can’t react fast enough to control the beam in real-time, so in a circular accelerator like the LHC, you have to predict what adjustments you’ll need to make for the next pass. Even still, it only takes particles a few microseconds to complete their 27 kilometer (16.7 mile) lap through the accelerator ring before they’re back in front of you — so there’s no time to waste.
But these incredibly powerful beam control systems, which Daniel says aren’t completely unlike the deflectors used to control a CRT, can be at odds with the scientific goals of the project as you also need to detect and measure exceptionally minute signals. Keeping all these systems in balance is a major challenge at CERN, as is compensating for environmental issues. A vast array of variables need to be taken into account while making measurements, from the phase of the Moon the the water level in the nearby Lake Geneva. Large global events, like the recent earthquake in Turkey, could be enough to halt observations temporarily.
Of course, as interesting as these technical details are, the topic of conversation for this Hack Chat was specifically “Life at CERN”, so of course folks wanted to know what an average day for Daniel looks like. Working in what’s called an equipment group, Daniel and his team are responsible for developing and building hardware for the accelerator. But since it runs 24/7, there are only occasional windows to actually get in and work on their new gear. Accordingly, an incredible amount of testing and debugging is done before it’s actually time to install the hardware. When each hour of downtime costs a quarter of a million dollars, due diligence is the name of the game.
When the accelerator is up and running, and directly working on its systems isn’t possible, the teams have the time to pursue new designs and experiments. But should something go wrong with the accelerator, it will immediately become the top priority. While there’s an emergency standby team that can be onsite within 45 minutes, they’re only capable of handling a small subset of possible problems. For anything else, system-specific experts need to step in. As such, Daniel says you need to be ready to respond to a call from the “Big Lady” at any time, day or night.
Often it’s some software issue that can be resolved with relative ease. But not always. Daniel recalled a time when he spent three hours diagnosing a system, only to find that a lightning strike a kilometer away was enough to upset one of the sensitive pieces of hardware.
Solving problems on a machine with hundreds of thousands of systems requires you to not only have an encyclopedic knowledge of how it all fits together, but the ability to quickly pick up on the smallest of clues. He describes it as “very high caliber hacking” — an apt term if we’ve ever heard one.
But he also notes, almost wistfully, that the LHC has been performing with remarkable stability as of late. When first brought online in 2008, he says the mean time between failures (MTBF) of the LHC was just five minutes. Now he goes months without hearing the siren call of the Big Lady, which gives his team more opportunities to work on smaller projects and machines. It might not be as exciting, but it’s certainly productive.
We want to thank Daniel Valuch for providing such incredible insight in what it’s like to work at CERN, and specifically on a machine as incredible as the Large Hadron Collider. We’re glad to hear working in such an environment is as rewarding as we hoped, and wouldn’t be surprised if readers looked at the sciences in a different light after reading his first-hand accounts. As always, the full transcript of this Hack Chat is available on Hackaday.io, which we’d strongly recommend checking out if you’re even remotely interested in high-energy physics or the nuts and bolts of the LHC.
The Hack Chat is a weekly online chat session hosted by leading experts from all corners of the hardware hacking universe. It’s a great way for hackers connect in a fun and informal way, but if you can’t make it live, these overview posts as well as the transcripts posted to Hackaday.io make sure you don’t miss out.
9 thoughts on “Daniel Valuch Chats About CERN’s High Caliber Hacking”
“you have to predict what adjustments you’ll need to make for the next pass” …… If I can do this, can I get a job with CERN? (LOL)
Only if you can pass the background and security checks or know somebody really high up in the food chain with hire/fire privileges. Actually knowing things, not so much
I’ve actually been screened by Interpol, so my background is clear, other than a parking ticket which got me a criminal conviction. Pah ….. I dont want to work for CERN anyways.
I don’t know of any jurisdiction in which a parking fine is a criminal offence. Please expand?
I’m in the UK and there’s a very obscure law that states that driving a vehicle more than 15 yards off the public highway is an offense. I was actually parked in an unofficial local car park used by locals to walk their dogs etc, even the local council agreed with me, but the magistrates, God bless them, decided it was not a car park cos “Justice must be seen to be done” ie we dont give a f**k about the law as long as what we do ‘looks’ ok. It was an educational experience and, whilst defending myself, I got to play the part of lawyer for about 4 hours, which was quite fun.
PS. Fined £10 !!!
Including costs? Quite the bargain, for four hours court time!
There’s no background and security check, you need a clean criminal record that you get from your local authorities. Like other jobs, you need to have a skill that’s on demand and pass a series of interviews. Been there, done that, got a 5 year contract.
How much is the bribe to clean your criminal record?
Please be kind and respectful to help make the comments section excellent. (Comment Policy)