Earlier this year a simple indicator LED brought the Keck 1 telescope, a 370 tons mass, to a halting stop. How exactly did an LED do this? Simple: it did nothing.
As it so happens, [Andrew Cooper] was just about the leave the summit of Mauna Kea (in Hawaii) when his radio instructed him otherwise: there was an issue. Upon returning, [Andrew] was met by a room of scientists and summit supervisors. “Yeah, this was not good, why are they all looking at me? Oh, h%#*!” The rotor wasn’t moving the telescope, and “no rotator equals no science data.” After being briefed on the problem, [Andrew] got to work. Was it a mechanical issue? No: manual mode worked quite fine, also indicating that the amplifiers and limit switches are functional as well.
Jumping from chip to chip, [Andrew] came across an odd voltage: 9.36V. In the CMOS [Andrew] was investigating, this voltage should have High (15V) or Low (0v) and nowhere in between. Judging by the 9.36V [Andrew] decided to replace the driving IC. One DS3632 later, nothing had changed. Well, maybe is one of the loads pulling the line low? With only two choices, [Andrew] eliminated that possibility quickly. Likely feeling as if he was running out of proverbial rope, [Andrew] remembered something important: “the DS3236 driving this circuit is an open collector output, it needs a pull-up to go high.”
Reviewing the schematic, [Andrew] identified the DS3236’s pull-up: an LED and its current limiting resistor. While the carbon composition resistor was “armageddon proof,” [Andrew] was suspicious of the LED. “Nick, can you get me a 5k resistor from the lab?” Hold the resistor on the pins of the chip and the amplifiers immediately enabled.
[Andrew] summarizes things quite well: “yes… One of the world’s largest telescopes, 370 tons of steel and glass, was brought to a halt because of a bad indicator LED”. It stopped things by doing nothing, or rather, by not turning on.
We love it when we get troubleshooting stories, and if you share our interest in problem-solving, check out this broken power supply troubleshooting or learn what could go wrong with I2C.
Edit: Keck 1 is one of the largest optical telescopes in the world. Thanks to [Josh] for noticing our error.
Bucket list Item #5
Visit the telescopes on the top of the Big Island.
If you wait long enough, there’s due to be a bigger (newer) one I believe?
You mean the one that just got approval to be built on a locally sacred mountain?
I’m fairly certain that to the myriad of peoples who have walked the Earth since the dawn of our species have claimed every iota of land sea and air as sacred. As long as it’s not a strip mall or garbage dump I think Pele will be happy to see mankind look upon the heavens.
Build a telescope on top of Harvard. Nothing sacred there. ;-)
[Ostracus], and to think it started as a Divinity School!
Telescopes get build on Hawaii for three reasons:
#1 – No air pollution
#2 – No light pollution
#3 – Steepest mountain on earth (you can bike down from the top at the observatory and will go through 7 climate zones in 1.5 hrs if I remember correctly)
If you’ve ever lived in Hawaii, “no air pollution” is a bit hilarious – you’re right beside an active volcano.
One of the main reasons for choosing Mauna Kea is actually that it’s in the middle of the ocean, isolated. So the air masses around it are dry, and much more stable than average, allowing for adaptive optics to focus really well.
I heard that the only concept Americans can relate to in regards to sacred sites is the Wailing Wall in Jerusalem, imagine building a telescope there! Why Americans care about a wall in Israel so much I don’t know.
Q) So what’s the difference between appreciating the spirits that live in sacred places and appreciating the beauty and knowledge of our universe?
A) The universe REAL spirits are MAKE BELIEVE
@Ostracus, does Harvard not have a telescope? It seems like they should.
@kCutter, This American would be just fine with a telescope built on the wailing wall. I’d much rather see a nuclear waste dump covering that whole hill so that nobody would fight over it any more. Then again, they probably would anyway because that’s what religion does to people.
Religion is outdated and should be overcome. Although we sometimes have similar problems with environmentalists. :-)
Scratch that one from your list. You can’t visit the telescopes at the top of the Big Island. They will only let you go as far as the visitor’s center and even then they won’t let you look through the telescopes at the center because they stopped that program as well. I was there last year and, as you can imagine, quite disappointed.
Also, who puts an LED in a pull-up circuit?
You can indeed visit the scopes on top. The road is public to the top, though you should have a 4WD vehicle and know how to use engine braking on the way down. Here at Keck we maintain a visitor gallery where you can go in and look up at Keck 1.
I got caught by this exact thing. You can’t use a LED as a pullup. If you’re using the signal driving the LED somewhere else, and, for reliability reasons, it has to be the same signal that drives the LED, put a 10k resistor in parallel with the LED.
Now, the voltage pulling the output high will be the full bus voltage, no LED drop involved.
some engineers think they are smart when they try to combine functions of parts, and then dont consider failure modes.
This is the same mentality that says “meh, should have used a 555”.
Thought this was the same mentality that says “meh, should have used something better than a 555?” MCUs usually have a whole lot more safety features that can be used than a 555, which is a single oscillator, more or less?
What? Like an LED that indicates “BAD LED HERE”?
Blue LED as actual indicator LED and a red led in parallel as a bad LED indicator LED.
Oops, the other way round of course…
There are Volvos that have an ‘instrument panel warning light burned out warning light’. Literally an indicator light that will illuminate if another indicator warning bulb has burned out. It won’t tell you which bulb is dead, but at least 8t informs the driver that they may be receiving a warning that is not being indicated. We used to love riffing on the warning light warning light in our friends ~mid 80s(?) Volvo. Yes, the body was painted in that classic Volvo gold color.
This. ^^^^^
Never combine a UI (e.g. display) element with a low level functional element unless there is zero chance of interaction. If space was at a premium, the suggested bypass resistor (sized to allow the circuit to work with the LED functional, open circuited, or short circuited would be a good solution. A buffer in a small SOT353 package driving the LED might be called for in other domains.
For completeness: my application was a LED indicator on a switch input to some logic on a circuit board. I wanted the LED to light when the switch connected the logic input to ground, and figured when the switch was open, the LED would pull the logic input up to the supply voltage. Wrong!. Even when there’s almost no current flowing through the LED, there’s a voltage drop. The logic input was seeing less than its high threshold and the circuit wasn’t working. A 10k across the LED did the trick.
Well, it is pretty rare, extremely rare actually for an LED to fail (at this point 1000 people will jump on me with their stories about LED failures). I have yet to see an LED fail that had not actually been abused, but I am sure it happens.
But you could argue that this circuit was a bit too clever. Using the indicator as “dual duty” to be a pull up and an indicator. I don’t think I would have done things this way, but given how reliable LED’s usually are I can’t throw too much mud at it.
Just another component failure and a bit of routine troubleshooting.
The write-up says this LED was likely from ~1990 when the boards were built. The boards are probably powered up continuously.
LEDs from the 90’s might not have been as reliable.
maybe not but if the assumptions that the board is from the 90’s and it was run 24/7 since then are correct, this led will have run for more than 200k hours… most led nowadays are rated for 60-80k hours so seems like a pretty reliable piece of equipment but of course ymmv….
90’s eh, I bet every single electrolytic capacitor was replaced at some point.. many times.
The 20mA indicator LEDs normally don’t get significantly warm, they last “forever”. Of course modern high power illumination LEDs at several amps get really hot and are good for 10000 to 50000 hrs.
LEDs from the 80’s and probably 90’s *were* less reliable. In any case a LED die can suffer metal migration just like a bipolar transistor.
@Ostracus – Walk into an electronics store today and buy an LED. I bet it was made in the 90s! If not the 80s. You must be young. The 90s weren’t that far behind today technology wise!
From the 15V CMOS logic, I figured it was much older.
When you have an LED with a ~130,000 hour lifetime and leave it on for 13 years, the display on your old microwave stops lighting up.
Having spent some time studying LED failure (after a long-running and ill fated interaction with a Chinese plant that just couldn’t stop their new InGaAlAs junction process from falling apart over a few hundred to a few thousand hours of power-on state. Think of all the early green LED traffic lights with sections of flickering LEDs)…
Some are sturdier than others, and the best should be nearly as good as a power diode. But defective junctions do happen, testing is generally not looking for the telltale signs, and they’re hard to spot anyway before they [very slowly] come apart over time. They’re also more sensitive to abuse and are generally running closer to their electrical and thermal limits.
The biggest problem with using an LED instead of a signal or power diode is that the dedicated diode is designed to be as close to immortal as possible. LEDs are not– they’ve got a rated lifetime for light output and are generally engineered accordingly.
Oops, actually it must have been a GaN process… it was about 12 years ago when green/blue were starting to get really cheap.
Ahem:
**One of** the world’s largest **optical** telescopes.
-Josh
With that much money involved you would think they would have redundant circuits and be able to eliminate problems of components failing.
You wouldn’t put redundancies in an interlock though. A single fault in a safety system should have an effect. Not knowing the details, stopping the drive until the fault is determine is likely the best way to prevent damage.
Many safety systems do in fact use redundancy in interlocks/trips. It’s common to use “2 out of 3” voting, so that a faulty signal does not unnecessarily disrupt production.
It’s obviously a conspiracy.
https://www.youtube.com/watch?v=VTUrdizRZyw
OMG OMG OMG, gubbmint have like 19 secret hammers for every ONE they admit to, we’re gonna get hammered!
For most of the government space science stuff I’ve worked on the budgets suck. You’ll be using super old stuff, getting spares from around the world as they shut down the other sites since the parts are out of production. He did a good job and should be commended, but I’m sure its just another band aid with no upgrades planned or budgeted.
A scientist from an observatory in Chile joined a vintage Macintosh e-mail list I’m on. The reason was they had a small radio telescope, a twin to one in the USA. They were built in the 1990’s.
The signal processing (and possibly operating the telescope) was done using a custom NuBus board. The scientist didn’t know if any other old Macintosh would work with the board or software so he joined the list to search for a replacement IIci motherboard. He found someone with a good board to sell, presumably he got the computer fixed. Several people suggested he get someone to figure out how to operate the scope with an up to date computer, and price it out so it’d look like a better deal than keeping the old dinosaur going. Push the benefit of being able to handle much more data and process it faster.
Another time someone joined looking for a specific old Macintosh motherboard to repair his CNC punch press. The NuBus boards and software had been made specifically for that model, accept no substitutes. It was from the press manufacturers lower cost product line. The top line models had fully custom built controls.
Funny enough, MAME emulates the Macintosh IIci, and also can emulate individual NuBus cards. There’s no reason why, given a working example, a developer wouldn’t be able to implement those cards in MAME as slot devices, which could then be used in an emulated Mac IIci, with any external data being supplied via the input/output systems that already exist in MAME.
The other year I sold a Radeon 9600 to an academic who had some favored old molecular modelling/visualisation tool that only ran on 9×00 class Radeons.
Our budgets at Keck are notably better than that. We have a whole new drive control system in plane and undergoing final testing. We should be using it regularly in a few more months.
I do admit to sourcing obsolete parts on eBay once in a while :)
Given how sensitive telescopes are to light, I’m surprised they didn’t already have some mechanism to ensure the LED wouldn’t be passing any current (and generating stray light) when the telescope is in active use.
As it is an “indicator” it may be in the Control Room away from the optics.
I’m going to disagree with most people here. If it was important for the LED to be on when the circuit was enabled (safety reasons maybe?), it makes some amount of sense for a bad indicator to also disable the circuit. If you just put the LED in parallel, it could provide a false indication.
An indicator should report the state of another part, not be an integral part of it’s function.
It reported it’s own failure for anyone really paying attention :-D
I can confirm the ‘dead indicator light prevents powering on’ acting as an intentional interlock on an old x-ray diffractometer I have to keep in working condition. The circuit around it is actually designed in such a way that if the voltage drop across the indicator bulb isn’t in a fairly narrow margin when it’s supposed to be lit, the high voltage generator for the x-ray tube won’t turn on at all. Bit of a bitch to troubleshoot since the service manual doesn’t mention that little quirk anywhere.
That’s a great point. After thinking about it, I agree with you over all other posts. If it’s a “motor on” indicator, it is safest for the motor to stop functioning if the indicator for motor function does not work as well. In fact, it is arguably the safest design. Imagine powering on a high power motor, not seeing the indicator light saying that it’s running, and then telling someone to enter the maintenance area where the motor is in fact running. Obviously there should probably be additional safeties against doing such a thing, but clearly it’s safer for the system to function in this manner.
No imagination necessary as this was actually controlling a high power motor in a human-accessible area :)
I also believe there are better ways to do it, an LED in a logic-level pullup may not be a robust design, etc. But I (and you) can at least see where the circuit designer may have done this intentionally, to prevent a situation where the LED was off but the amplifiers were enabled.
This seriously makes me wonder why they have custom circuits driving a very expensive telescope? Anyone here surely knows custom is more fun, but in real world situations with money/grants on the line a swappable IO card or VFD seems much faster to fix. 1) Swap the cards, 2) Get the big machine running 3) Troubleshoot the old card while the big machine is running and making money (or science data)
if you wait long enough, every circuit becomes custom by obsolesce. look at all the ancient computers still running for example
You know, this idea might blow your mind, but I was thinking..what if, and hold your hat, what if people making custom stuff were allowed to also make swappable boards!
Of course we’d first have to convince the UN and the world’s supreme courts, but imagine!
:)
They do use off the shelf motion controllers and servo amplifiers. The custom electronics looks like is supervises the whole lot and ensures it works safely. Last I checked the Keck was using a Delta Tau PMAC Lite (http://www.deltatau.com/DT_Products/ProductDetailDescription.aspx?CatID=100-PMAC%20PCI%20LITE).
A diagram of the system can be found here: http://irtfweb.ifa.hawaii.edu/~tcs3/tcs3/Design/T3-3000_Telescope_Servo_Overview.pdf
Now they could do all that with a BeagleBone Black running Linux CNC.
because “not invented here”
The old drive system has a lot of custom parts, including the logic boards and the servo controllers. The new system is almost completely off-the-shelf parts with a couple custom PCB assemblies that are mostly just interconnection to wed everything together.
Or bench test parts before installing them.
Much how the generator in older cars stopped working when the battery indicator lightbulb was broken, because the excitation voltage ran over it…
True. But technically, it did not stop working, it could not start up. If the engine is running and the generator already produces some power, this auxiliary excitation is/was no longer necessary
Do newer cars fix that bug? I wouldn’t count on it!
Not the largest telescope….
https://en.wikipedia.org/wiki/List_of_largest_optical_reflecting_telescopes
I love it! the Effective Aperture is listed in meters, and the Aperture is listed in inches!
Just cover it with black tape
So then, a queen sized bed sheet is out of the question?
Maybe the led would “bright at least 80%” by the “end of its life”, but how long until it would dim to a halt completely?
LEDs like many things fail in all sorts of failure modes
They could slowly short, slowly open, short without warning, flicker
I got caught in the same thing for the paralleled outputs of several comparators that was failed from the beginning
The most embarrassing part was that I rebuilt the entire circuit as I ran out of ideas only finding the fault years later when the LED failed on the new one
Regardless of the technology issues involved in this troubleshooting, there is also the biological factor.
I visited the visitor centre below the summit and observatories some years ago for the usual touristy evening of talks and small telescopes observations.
Something I remember apart from ‘beware of invisible cows’ (you’ll have to google that)
was the effect of and need for acclimatisation due to the altitude and oxygen levels at 14000+ ft
One of the guys speaking described colleagues keeping a careful eye on each other when they visit the summit for the first time or after a period of absence.
The altitude can affect people in different ways, apart from the potential health issues that can start to affect people even at this relatively low altitude there are also the potential cognitive effects.
He described instances where people might forget things, in his case, he was always losing his keys, and because of this kept them on a spring loaded recoil wire attached to his belt.
But others lost pagers, tools, notebooks, drinks, two-way radios, and rare for the time mobile phone.
He described one incident in particular where a colleague had been asked to go and make simple repair to a support or frame of some kind involving sizing, cutting and fixing a length of steel.
After several hours he had not returned and although he was visible (so they knew he was ok) they were not sure why the job was taking so long. When asked how he was doing, he (allegedly) replied, ‘you know, I’m having trouble here, I’ve cut this bar three times now and it’s still too short !’
The task/problem may be simple, but the environmental conditions may also effect your ability to solve the problem efficiently.
Ho ho. We had the same problem with a massive analogue mixing desk intended for outside broadcast. One of it’s major modes of operation was Transmit, which locked out certain functions and routed all sorts of signals all over the thing. The driver for the Transmit mode line was also open collector, but the pullup was a number of indicator lights which were incandescent bulbs. During testing we had the console running with only one light plugged in, and the globe blew. Suddenly no Transmit mode! I found the fault exactly the same way, a strange voltage on the line. I installed a 1K pullup to prevent that ever happening again,