This repair/tutorial video by the telephone Connections Museum of Seattle features an amazing piece of electro-mechanical technology from the 1950s — the 5XB trouble recorder. Museum volunteer Sarah the “Switch Witch” has a deep passion for old phone equipment, and gives an excellent description of the trouble recorder, the problems it solved, and how it works, and how they went about fixing it.
As central office switching became more complex and more dense, the manual methods of hunting down faults became unmanageable. Semi-automatic approaches using trouble lamps, but even that had its limits. This “stack trace”, which could have hundreds of indicators, had to be frozen while the technician recorded the status on a form. If another fault came along during this time, it was lost. The solution, using the available technology of the day, was a mind-boggling punched card apparatus that punches over a thousand bits of information when an switching error is detected or when various watchdog timers expire.
