It happens to the best of designers, spending untold amounts of time designing a complex device just to find out that you missed a trace, or you couldn’t rout something to something else. As time marches on its becoming a bit less common to pop open a commercially produced device and see a little jumper wire or 2 flying across a pcb, or a resistor straddling an IC.
But when [Ilektron] opened up a Yamaha Dolby Pro Logic receiver to scavenge for parts he saw a very big “oops” and a even wilder fix. The maker took a pair of relays, flipped them “belly up” and hot glued them into place on top of a pair of ICs. Then the mess was “dead bug” wired to the circuit using insulated and uninsulated bus wire, contacts were then reinforced / insulated using more hot glue.
This is one of the most hacky fix we have seen in a commercially produced product, but we would love to hear your amusing horror stories of “WTH did they do?” So join us in the comments after the break.
Honestly, it looks well done. It seems to be well laid out. The soldering appears to be good. The parts are secured well. Wire color seems to have been respected.
Yes, it’s indicative of a major engineering oops, but really, in the grand scheme of manufacturer fixes, this one rates pretty highly in my book.
Maybe a result of some part shortage (if the pcb had holes for other relay’s)?
Canadian here
bought a 400 watt ******STER inverter from ******** tire. Not exactly the highest quality in the first place< but should at least be safe for the everyday consumer.
took it home, connected it to a 12v battery, and nothing. waited for a second… then smoke everywhere. so i disconnected it, and cracked it open.
traced everything back and found that it had a couple fuses in it. in parallel with the power rails, WHICH HAD THE WRONG POLARITY ON THEM. So the device tells you to hook the battery up wrong, then burns itself into a pile of slag right away, because the fuses are improperly placed on the board and thus cannot do their job. Smart people.
what i found the most amusing, is that it had an aluminum heatsink on it, covered in twisted edges and stress marks. took me a minute to realize how it got like that, then i realized that they took the barstock they wanted to use, and instead of cutting it or machining it (as would even your lowest end chinese factories)to the appropriate length…. they hammered and bent it in a vice back and forth until metal fatigue broke it off- and they didnt even bother to knock off the edge.
and oh god.. the thermal paste… eeeeeeveryywhere. in the switches… in the display… eeeeeverywhere.
posted pictures and a full explanation to ******** tire's website, and asked for an explanation. they responded by ignoring me, deleting my post, and my profile.
I love it when they put the fuses in backwards!
I love it when people don’t realize that fuses on the hot line coupled with reversed polarity input = zap. (eg under normal circumstances, the fuses would blow to protect hot-to-ground shorts, but if the DC input was reversed, they would do absolutely nothing)
I had a Yamaha Surround receiver that had several dead Asian roaches in it when I opened the box.
Does that count as dead bugs?
When I clicked on the news I actually expected something along this line :D
Same here actually, when I opened an OLD computer once, full of bugs and carp
I blew 22 million last week because of a bad database commit. Not my fault directly, but whoops!
Care to repost the pics? I’d love to see them.
Yeah, me too, wanna see pics! :)
Also: why did you star out the maker’s name? Let them be known for their crappy quality!
Purchased a PSone from a pawn shop, opened it up to mod it, noticed a chip glued down and wired in dead bug using wire wrap wire (soldered mind you). Thought, “Heh, that’s in my way…”.
Outcome: It was already modded and I very nearly unmodded it by accident.
Actually, it quite possible that this was done by a repair shop or as a factory errata as mentioned above. The dead bugs could have been there for many reasons.
The mod chip… hehe, nice luck there.
I used to work with a girl who had to put the specs from data sheets into the CAD package.
She got an IC the wrong way round from top to bottom view. As it was a few hundred pins of SMD chip on a large complex board with lots of other stuff already on, there was nothing to do but bin the few boards that had been finished. The “mistake” cost the company more than her house was worth. (about £300-500K about 15 years ago)
They were surprisingly calm about it “we’ve all done something similar, don’t worry”.
I think on average she’d made less mistakes than predecessors, although the one she had made was the most expensive. She definitely didn’t make that mistake again!!!
When I started as an engineer at HP I had a senior engineer tell me once that until you’d made a mistake that cost the company hundreds of thousands of dollars you were a rookie.
I’m dating myself but I started designing digital logic before the wide-spread use of ASICs. It wasn’t uncommon to have a board with hundreds of chips on it. We had rules about how much rework a board could ship with. The manufacturing techs were artists with implementing the rework. Very neat.
“Don’t blame it on the girl” (Where have I heard that before?)
If mistakes have such serious consequences they should be checked and double checked by at least 3 or 4 different people.
If not, then it’s the fault of the management, and not the girl. Besides that, that’s what prototypes and production pre-runs are for.
Did anybody think of bending the pins of that smd chip and solder it in upside down?
I actually worked on a device once that in order to install the off the shelf part I would have to mount it on the back side of the circuit board. They had the IC manufacturer bend the pins toward the label side. One of the nice things about ordering batches of ICs, I guess. Did not know what the IC was until I had it out. Also seen a lot of field modified boards as well. As in unit was installed and powered up and went boom. Units where not tested when made. The just made them and the start up techs got to make them work.
or maybe yahama designed the board just as the relays arrived and went to install them and went “oops the relays are bigger than what we thought and wont fit the board”.
no problem we will just dead bug install them.
i have seen on pc monitor by mag that there was some kind of voltage regulator dc to dc converter board about 3 inches square attached by 4 plastic clip on risers to the inside of the case
you will recognize it if you open up a monitor by mag you will see a board attached to the inside of the case with a heat sink a transformer coil and 4 wires leading to the main board.
i guess mag found there was not enough room for the unit on the board so they made a separate board (cheaper than re printing all new main boards)
Years ago the company I was working for had contracted an OEM to design and build an eval kit for our SoC with all the bells and whistles. It took 7 iterations of the design to get something we felt comfortable releasing. Among the WTFs were big fat caps to ground on each of the resistive touch screen lines, CCFL backlight inverter on the same PCB as logic without much in the way of isolation, and not releasing CPLD code to us because parts of it were secret sauce (it was an implementation of a parallel port Wiggler, which IIRC was just a couple of latches).
But the one that got me was when they changed LCDs mid-design. New boards were being produced to match the new connector and pinout of the LCDs, but there was still considerable stock of the old style boards. To save some money they build up a bunch of kits with the new LCD plugged into the new-style connector deadbugged to the pads for the old one as the pincount was the same but the pitch slightly different, and copious amounts of hot glue covering the atrocity. The cable, which had all white leads, had pins rearranged by hand in the shell to match the new pinout. These looked almost just like the straight-through ones for the newer kits…
It would be barely passable on a cheap consumer product, but this was meant to be taken apart and to represent our chip in the best light. The worst was that we were charged the same per unit as the newer builds…
The comments are really interesting. It is neat to see how people/companies respond to the “Oh crap… I screwed up”
I remember 20 or so years ago, ollivetti sending out a ‘fix’ for their XP7/9/11 PCs, consisted of a single wire and a sheet of paper describing the solder points for you to perform the fix yourself, fun times.
They provide you with a bit of wire but not a soldering iron? Classic corporate thinking :-)
I once designed a board with carbon button pads on it. It worked in testing and prototyping, but then when the box went to production I forgot to double check the exported dxf file to the designer. We ended up with a 10000$ mirrored box mould!
These relays do go bad, i have replaced them my self in this same fashion. So its possible it was a repair.
I agree that this was fixed by some third party how are you sure that it came from the factory like this?
I had to do a similar fix when my yamaha reciever lost all av video inputs i just used a resistor and cap to convert composite to s-video then converted it back to composite on the s-video output you can’t tell the difference in quality and looking at the board you can’t tell unless you look very close and its been working for about 8 years now!
This is pretty crazy: I have a Yamaha receiver, and had to replace all of the audio channel relays two or three years in because they all went bad, one-by-one. One day the, say, LH Rear channel wouldn’t turn on unless you cranked the volume loud for a second to pop the relay over, and they day after it wouldn’t work at all.
Good thing the insides of the cases are roomy – I’ve now got six new relays dead bug wired in to replace all of the stock ones.
There are several reasons why you might see boards that have jumpers, cuts, and kludged parts. In my career working for a small electronics companies I have seen the following:
Parts going obsolete unexpectedly.
Parts being so popular you can’t get them.
Parts being made unavailable due to some patent lawsuit.
Parts with flawed datasheets.
Parts that do not live up to the specs in their datasheets.
Yes we had our share of mistakes that were worked around with kludges too, but not all could be foreseen or eliminated by testing the prototype. Some were only discovered once in the field after a year.
Two reasons for the inverted relays:
1) coil clamp diode opposite to the PCB pinout.
2) coil polarity wrong [magnet in the armature]
I’ve been burned by both.
Of course, I did it myself, learning Altium Designer, and inverting an entire side of a chip. I don’t remember exactely why, it was a common TTL SMD chip, maybe LS245, but I did and the result was…er…”nice”.
I’ll post a photo of the “repair” later :)
When I was in the beginning of AVR design, I remember creating a jamma-to-ps2 keyboard adapter. I forgot ALL the pull-up resistors :) Ooops :)
This stuff happens in all industries. I used to make car parts in a major plastics factory. They make pretty much every plastic part of a certain brand of foreign car. Keep in mind that some of these cars were up to 50% plastic by volume.
Because we got bonuses and/or free shares if we could apply some critical thinking and solve problems… There were plenty of kludges wherever we could get away with it. If a flaw was not deemed a safety or cosmetic issue, we were free to creatively repair the faulty parts. Among the doozies I’ve seen:
– Gluing or “welding” parts together if the molded connectors fail to hold, making the component almost impossible to dismantle.
– Cutting up dozens of coat hangers to get short segments of wire used to sturdy up plastic hinges (after drilling out the plastic pins).
– Replacing misprinted vinyl stickers with cheap paper mailing labels from the office.
– Taping pennies onto the back of weight-sensitive parts. A fancy cupholder assembly was supposed to glide open smoothly, but the production plastic was lighter than the prototype so it just went “thunk” until someone got the bright idea to balance it for about $0.02 per part.
– Shimming parts with everything from cut-up chunks of reject parts to cardboard boxes.
– Hand-carving useful tools out of waste plastic or other garbage. (Actually I’ve still got a very nice “bone” paper-folding tool that came from a piece of cut-off plastic sprue).
– Using short lengths of plastic pallet strapping from the shipping dept to hold things together.
– We had a million dollar machine to electrostatic-ally coat some plastic parts with a light plastic fuzz (flock). A shipment containing custom-fit electrical contact pads for the latest product didn’t show up… so I wedged a ball of tinfoil in there and mashed the product in on top of it. The resulting tinfoil contact ran about 2,000 parts without fail… and looked like a wounded tribble after some idiot ran the machine without putting the part in.
+5 points for the tribble reference.
I just told my dad and he said: “Oh, we used to do that a lot in the computers we sold back then… was it the Spectrums or the Commodores?… If you plugged a TV, they’d display white letters on black background, which was quite tiresome for the eye, and we wanted black on white instead… had to fit in a NOT somewhere, so we used to dead-bug them.”
Yes I’ve seen that a lot in old 8 bits, especially “revision 1” or “revision A” boards, when a new one came out, and died after 2 weeks, it would come back from warranty deadbugged, and probably for several weeks after, the ones in retail would have that fix pre-applied.
Thanks for all the great posts in this thread…loved reading about all the colossal screwups! (why is it that mistakes are more entertaining than reading about someone doing it right the first time?)
-greg
I showed this to my dad and he said: “We used to do that a lot on those old computers… were they Spectrums or Commodores?… Anyway, when you connected them to the TV, they had white letters on black background, which was tiresome for the eye. We wanted black on white instead, so we had to fit in a NOT somewhere – so we’d dead-bug it”
I designed a circuit using a SPDT analog switch IC. I found out after PCB assembly that the “D” on the pinout mean drain not digital and “C” means control not common (for the switch). This circut was repeated 14 times on the board!
20+ years ago a coworker made a mistake that destroyed a lot of data. So much was lost that it cost $80,000 in payroll to have it manually re-entered. My friend went to our boss and said “sorry, Boss, I really screwed this one up, I should probably resign if you don’t fire me.” The boss told him “Hell, you can’t quit! It just cost me $80,000 to teach you that lesson. I can’t afford to train someone else.”
Yamaha didn’t do this. Someone else did.
unit died and went back to seller, and seller used creative thinking and took unit for repair, and sold it again.
The most scarey thing in a lab…
an engineer with a tool in his/her hand.
Case in point, I was working in a battery plant testing cells. We would gang the cells, at several hundred pounds each, in a temp controlled 45′ trailer to test. We used a computer controlled load bank. We told the engineer on the project the load was wrong, and he just said ” I’m the engineer, do it that way”. When the switch was thrown, poof! The load bank went up in flames, with no emergency cut-off. $25K in damage just to the load bank. At least he manned up and took the blame to the head office.
Not a commercial screw up but a colossal one none the less…many moons ago i wanted to rack mount a piece of AV gear (commercial LD player) I figured no problem bent the ears on the brake drilled the racking holes pre drilled the mount holes and lined up on the LD player drilling thru the case(for proper placement of course) and right through the board just inside the case…out came the soldering iron and a chunk of 22 gauge wire…worked like a charm but I am very thankful the boss never found out!
I have a fire alarm/ smoke detector, with the polarity of the battery is mixed up.
If you put the battery in it, how they instruct you to do, it won’t work.
If you turn the battery around, it works.
(I am terrible at cooking that darn thing went three times off lol )
So always check if that thing works after putting the battery in it.
(there is usually a light blinking or a test button)
Your life or that of your loved ones can depend on it.
A not so funny bug if you ask me.
That reminds me, I have to contact them.
All this talk of correcting mistakes reminds me of my 1st year A-level electronics course work in school.
I’d made a 16 channel mixer with active eq all working through op-amps, but when I designed the board I forgot to remember that the pads would be back to front after etching, so I had to flatten out the through hole pins of the op amps I’d bought and SMD mount all 48 op amps (gain control hi low filter amps and output buffer stage) to the 16 different channel boards I’d made.
Fact is that everyone makes mistakes, it’s how those mistakes are dealt with that measures the person.
64, not 48, 4 chips per board, 16 boards.