This repair/tutorial video by the telephone Connections Museum of Seattle features an amazing piece of electro-mechanical technology from the 1950s — the 5XB trouble recorder. Museum volunteer Sarah the “Switch Witch” has a deep passion for old phone equipment, and gives an excellent description of the trouble recorder, the problems it solved, and how it works, and how they went about fixing it.

As central office switching became more complex and more dense, the manual methods of hunting down faults became unmanageable. Semi-automatic approaches using trouble lamps, but even that had its limits. This “stack trace”, which could have hundreds of indicators, had to be frozen while the technician recorded the status on a form. If another fault came along during this time, it was lost. The solution, using the available technology of the day, was a mind-boggling punched card apparatus that punches over a thousand bits of information when an switching error is detected or when various watchdog timers expire.

The trouble recorder in the Connections Museum was not quite working. But with a lot of patience and access to a service manual, the team eventually got it up and running again. Now the biggest issue now is getting new blank cards printed when the few boxes they have finally run out.

If you are interested in these kinds of intricate electro-mechanical systems, do check out the video below. We especially liked the mechanism that broke up 1200 bits into a timed sequence of ten each 120 bits to drive the punches using motors, cams, gears and relay contacts. You can read more about this trouble recorder in this Bell Labs Record technical report (pg 214) from May 1950 (interestingly, this issue leads off with Dr Hamming’s famous paper on error detection and correction codes).