Fail of the Week: Arachno∙fail∙ia

Going down the list (FCC, CE, UL, etc.) we can’t think of a regulating body that will test for this failure mode. Reportedly, a $1M irrigation system was taken down by a spider. And an itsy-bitsy spider at that.

This fail turned up as a quick image post over on /r/mildlyinteresting but I wasn’t the only electronics person attracted like a moth to a flame. Our friend [Sprite_TM] popped in to answer a question about conformal coating. Seems this board was sealed in a waterproof enclosure but was obviously not conformally coated.

fotw-spider-short-relay-diagram[Sprite_TM] also helped out with some armchair-engineering to guess at what happened. It’s not hard to tell that the footprint on the board looks like a set of mechanical relays all in a line. He looked up the most likely pinout for the relay.

We’ve superimposed that pinout on the board to help illustrate the failure. High voltage comes in on the pin shown with the red trace leading away from it. On either side of that pin are the connections for the low voltage coil which switches from normally closed (the pin in the upper right that is not connected to anything) to the normally open pin (which has the wide trace leading away from it).

So there sat the high voltage pin in between the coil pins when, along came a spider. It shorted the pins and presumably all the way back to the power supply for the low voltage rail. [Fugly_Turnip] (the OP) share some additional detail about the system and this failure; in addition to this card it fried the control module as well.

Another comment on the same thread shares a different story of two boards mounted next to each other with a bug shorting a 1/4″ air gap between two boards and causing similar carnage. Have you encountered Arachno-fail-ia of your own? Let us know below.


2013-09-05-Hackaday-Fail-tips-tileFail of the Week is a Hackaday column which celebrates failure as a learning tool. Help keep the fun rolling by writing about your own failures and sending us a link to the story — or sending in links to fail write ups you find in your Internet travels.

85 thoughts on “Fail of the Week: Arachno∙fail∙ia

      1. Waterproof enclosures aren’t always bugproof. An IP rating of 33, for instance is technically waterproof but allows holes that a spider can crawl into.

        In this case, however, the “small hole” could be anything from a redundant cable entry that wasn’t plugged to a small 1/4 inch hole drilled in the bottom enclosure (which returns the enclosure to IP33) on the theory that the more waterproof you make an enclosure, the harder it is for the water to get out when it eventually gets in. Both are very common in my experience.

  1. When I was 12-ish, I received a HF radio-telephone used by local car factory. It was huge, heavy, full of vacuum tubes, and broken. When me and my brother removed the chassis from the case, we discovered more bugs than in Windows Me. We ha literally a rain of bugs on the floor of our room. We didn’t know if they were the cause of failure, because there were two vacuum tubes turned into air tubes…

  2. I’m somewhat surprised at the lack of a conformal coating on that PCB. We have outdoor scramble pads from MANY years ago that have coated boards, then there’s then new exercise boards for our generators that don’t.

    1. Conformal coating aside, this unit would never pass certification where I am due to the pin spacing.

      Relays with that pinout are only really useful for around 50Volts or so (my opinion). Mains voltages have lots of nasty spikes so you can’t use parts that are rated at just above main voltages. At least 5 times the voltage is a good starting point. The same goes with your pin / track layout.

      If you look carefully you can see the traces on the other side and there is nothing connection to the N.O. pin so it would have been just as easy to use a relay with better isolation.

      1. Yep, as RÖB said,this unit likely wouldn’t pass the CE certification in Europe for the reasons he pointed out. At least it wouldn’t pass at an accredited emc laboratory ;) For 230VAC applications there must be at least 8mm pin spacing to low voltage parts of the circuit. Conformal coating doesn’t change that for you,but would probably helped in that case.

        1. The conformal coating improves the creepage distance. But the 8mm are only required, if safety isolation or low voltage operation is required.
          If the whole thing is in a grounded or insulated enclosure and you are not able to touch any part of the electronics, than that’s no requirement. Think of countless remote controlled sockets or timers with an internal supply with a capacitive dropper. Perfectly legal with the right enclosure.

    2. Came to comments with the same surprise; I worked on boards switch mains voltage that were in NEMA6 / IP67 enclosures and we still would get condensation (we typically traced it to line workers failing to caulk inside the conduits leading underground). Conformal coating went from a site-specific option to the default.

  3. Not Arachno-fail-la but ant-fail-la. In the South West US fire ants love electrical relays. Both central AC units and well pump relays are affected by this phenomena. The ants crawl between the open contacts and are fried by the 240AC juice. The now caramelized carcasses form a coating on the contact. After a few dozen dead ants the contact is insulated and the system fails. At this point you have to open the relay assembly and use an emery board or fine sand paper to clean out the contacts. As frustrating as it is for an AC unit to fail in the middle of a sweltering Texas night, its even worst to be in the shower when the accumulated water pressure in the pipes is spent, the pump fails to come back on and you have no more water. now you have to get dressed, go to the pump house, clean the (#*&#($ contacts and resume your shower.

      1. Yup. Is there some reason these other type of relays can’t have their holes sealed? Or put in some sort of suitable case, with maybe potting compound or the like to make them tight? Ideally a potting compound that ants don’t eat…

        1. Seems pretty common in older central ACs for some reason. My folks’ house in Illinois had a (70s vintage?) AC until recently. The relay for some reason sat on the conduit halfway between the unit and the house, with a separate coil and contactor unit and some openings in the bottom. Every few years we’d have to open it to scrape out the beetles and earwigs that are preventing the contacts from closing.

      1. Virginia is getting not just fire ants, but the so-called crazy ants. They have this odd behavior where they give off an alarm pheromone when they die, to let the rest of the nest know “hey, trouble is this way, come kill it”. Only problem is, they like to crawl into electronics, and the first one dies, then a few of it’s buddies, then half the nest decides to chew it’s way in to find out why the first group died.

    1. Do you think conformal coating would help or would the ants eat away at it or something?
      I’ve got a bottle of the stuff and could see it being an easy diy fix/prevention.

      1. Kind of need the exposed metal for the relay to work correctly unfortunately. I suppose you could seal off the relay somehow – but the ants will get in through any tiny crack. Apparently they’re attracted by the electricity.

    2. ” …the (#*&#($ contacts …”
      I was going to comment that you use perl-regexp in the text,
      and then I pressed the ‘Report Comment’ instead of ‘Reply’ .
      -another fail of the week…

      1. No. Neither of the boxes can be sealed in any practical form. The AC unit relay sits on a small shelf built into the AC unit’s overall casing and accessed via a small panel. To seal it I would have to remove the relay, remove the AC unit sides and to build a custom box inside the opening, use bulkhead fittings to connect the 240VAC and the 24VDC lines to the relay and then re-fit the relay to the new housing…. hmmmm…. it sounds like a hackaday project. The pump relay unit has a similar issue. Although it is a stand alone unit, activated by a push/pull lever that is part of the pressure assembly, it would need bulkhead connectors in order to work…..hmmmm another project….

  4. I see this kind of carnage daily at our service departement. We do gate automation. Spiders, snails and even mice find their way in ‘bad’ installations of gate control boards. Installers too of ten forget the importance of sealing their equipement..

  5. @Mark Smith: That was glorious! LOL! I was going to say something about a bug in the system, but your spider gets revenge on the waterspout comment is just too good!

  6. I still don’t understand who thought it was a good idea to put one of the HV contacts between de two coil contacts of this kind of relays. Or why. Can somebody enlighten me?

    1. Pretty normal relay pinout, I must’ve seen dozens laid out that way, and I’m not someone who sees a lot of relays in their everyday life. For SPDT it’s the standard I think. “HV” is usually mains, up to 250VAC or so.

      It’s probably a lot to do with the internal construction, if you look at a SPDT relay with a clear case you’ll see. DPDT ones are different, they group the 3 pins for each switch together.

    2. There are dozens of relay pinouts, some better than others at different things. Like an RF relay (yes, that is a thing) will want to be built with shielding and isolation in mind. Some relays as other have pointed out are open frame affairs with everything exposed, others are hermitically sealed. “Ross” is a company that makes things for high voltage work and they make some giant open frame high voltage relays out of G10 material with anti creepage slots milled all over them and contacts with rounded edges to prevent corona. This case is probably due to poor engineering and selecting the wrong part more than anything. And the lack of conformal coating backs that up.

  7. I was once experimenting with two valve regeneratve radio receiver, powered by 90V anode battery (you know where it is going). Then some noise came through my headphones. Next I noticed an ant shorting two pads on my PCB. Smoking ant, to be precise…

  8. Clearance only needs to be 4mm for line voltage. And you wouldn’t need to conformal coat the PCB if it’s in a sealed(!) enclosure.
    Conformal coating is a PITA for repair of boards, and you wouldn’t want to use it if you didn’t have to.

    The problem is that fried bug is conductive, so the carbon track that formed from line to DC didn’t help matters.

    1. Unfortunately it is not a simple as thet. “Clearance” as you call it has many factors voltage, altitude, humidity, temperature, condensing temperature.

      Even the voltage needs careful consideration. You can’t design around line voltage (110/220/240Volts) because spikes can easily go over 1000 Volts. You need to consider failure modes.

      Then when you have condensing humidity or dew, no conformal coating will save you because the condensation will act like a capacitor shorting out AC without making any contact.

      In a more controlled environment you can just “do the math” and be done with but mains/line voltages are far from predictable due to environmental factors.

  9. That looks like the relay board out of a Lindsay/Zimmatic FieldBoss panel. How do I know? I service them for my $DAYJOB. Those are expensive machines, but they are hardly $1 million. The whole control panel is in the realm of 4 to 6 thousand. (That isn’t going to do damage outside the control panel.) The full machine is in the ballpark of $60k.

  10. Dont know if many of you gents play with automotive stuff but even vircuitboards inside absolutely watertight enclosures are conformally coated. Atleast in decent cars. Makes rework a little harder but very greatfull of it when you see the mess some cars electrics(french mostly) get into.

    1. Most of the stuff won’t be reworked anyhow. The conformal coating there is to prevent metal particles from shorting out anything (0.5mm pitch IC pins). The metal particles can remain from the manufacturing of housing components

  11. In the early 80’s, I encountered a 386 computer that died due to a mouse problem:

    One of the metal tabs that block unused card slots was missing, and a mouse took
    up residence in the computer case. The mouse did it’s thing, and the conductive
    liquid it emitted proceeded to short out the motherboard.

    1. There’s also snakes in computers as well. Doesn’t need a missing cover, a standard opening like the one for power supply is enough for a small snake to slither in and sizzle to death.

    2. 386s didn’t come out before 1985… you probably mean 90’s.
      Sorry for nitpicking. I have a small computer museum… The oldest 386 I’ve encountered was made in ’88.

  12. My church has commercial electrical service, with I believe 4 phases into the building (our master electrical panel has 4 large breakers, and each one branches off into its own subsidiary service panel). We have a commercial kitchen with walk-in fridge, as well as an industrial-strength HVAC system, so we have a lotta juice coming in. A few years ago, a sizable portion of the building lost all power. When the electrician came out, he found that a snake had crawled into the big green transformer box outside and had grounded itself to one phase, blowing its master fuse and cooking the snake. The temporary surge led to an interesting failure mode within the UPS that served the server and phone system: it caused the standby relay to weld, thus necessitating total replacement of the UPS. 2 weeks later, that failed in the same way, though not due to the same cause. APC told me they had gotten a batch of bad relays that were prone to welding. The 2nd replacement has not yet failed, after at least 3 years.

      1. A church I know had a problem with mice eating the (line voltage ~90v?) speaker wires on the long runs across the building. Apparently still enough voltage to kill a mouse sometimes.

  13. Many years ago I was working in a Melbourne TV transmitter station and was told of a fault the had a rhythmic fail mode. Every few minutes it would drop off then restart. It turned out there was a Huntsman spider ( http://australianmuseum.net.au/huntsman-spiders ) in the works and it would touch something and cause the overload, but the shock would make it lift its legs up. After while it would tire and lower the legs…
    The fault was fixed with a wack!
    These spiders love the inside of houses and we often see them on our walls. As they can grow up to 15cm across they do look pretty alarming, particularly to those afraid of spiders. They are generally considered pretty harmless to people and when I spot one I invite it to leave for the vacant block next door.

  14. Add an acetylcholinesterase inhibitor to the bottom of the enclosure. You could use 1,4-Dichlorobenzene, except it is a carcinogen.

    Oddly enough the safest yet effective option may be the essential oil of rosemary in one of those slow release jell blocks, just make sure it isn’t going to dissolve any of the plastics you have in your parts or enclosure.

  15. At a plant I was working in, we had a mouse go phase to phase inside a 600v MCC. It blew a great big hole in a copper bus bar and blacked out the plant for most of a day.

  16. I have a few more memories,…
    30+ years ago I was working at the Radio Australia transmitter site in Shepparton. The transmitters were 100KW on the HF bands. Occasionally birds would land on the feeders and burst into flames and start grass fires. Often all that was left was a couple of legs with the feet firmly clamped to a line.
    Another time a cat got into the aerial switch and caused a lot of damage.

    But my best memory was of a bird getting into the transmitter hall. The transmitters were open at the top and the bird flew into transmitter one. There was a loud burst of program (the sound of the modulated arc) and a bang then the transmitter overloads dropped out. Amazingly, the bird flew out but the poor thing was not in the best of condition. I forever have the vision of this bird flying along the transmitter hall, trailing smoke and loosing altitude until it crashed to the floor. The bird was put outside in the garden and was gone later in the day. I hope it survived.

    There are some pictures of the place here..
    http://bpadula.tripod.com/shepparton/index.album/1973-antenna-switching?i=13

  17. Only somewhat relevant…
    I once had to open up a bunch of exterior speakers to service…Mud wasps had filled every screw hole in with mud flush to the surface (and these where many deep 6″+ holes). I dulled a few drill bits drilling the mud out so I could get a screwdriver in.

  18. Probably many of these failures are just soldering fails. There is so much frass (bug poop) in small spaces that are open it’s pathetic. Cockroaches, ugh! When it gets wet though? What happens? The relay points is real though.
    Seal?… pinipeds! Otherwise not happening for long.
    The only sealed relays I know of are mercury relays and reed relays which usually have little current rating but are very good on low resistance and have glass sealed leads. The so called plastic “sealed” relays are still subject to all the crap in the air where there mounted, plus the outgassing of the plastics and glues that “seal” them.

  19. A wasp took out my AC. Shorted across the run capacitor for the compressor and the bugs body was low enough resistance the compressor would fail to start. Only hum. I figured it was toast until I took it apart and saw the blue burnt bug body! Removed and the thing sparked to life again.

  20. I once got a tool back to the r&d with a bug problem (only the emi filter wasn’t coated on a prototype). We ended up the day catching and frying flies and ants to see how much damage it could do. For information, a dead fly starts moving its legs and wings at 30V and smokes instantly over 60V.

  21. Good Grief author, please restrict your commentary to which you have a professional knowledge of… This reminds me on all of the Li battery failures for laptops, aircraft, and boards. There are standards and directives in place that, had they been properly evaluated and assessed, would have prevented these catastrophic failure modes.

    “Going down the list (FCC, CE, UL, etc.) we can’t think of a regulating body that will test for this failure mode.”

    Cannot think of any product safety standard that does not specify enclosure requirements, abnormal operating conditions, Pollution Degree, O/V category, insulation coordination, PCB construction, component and conductor spacing, blah blah blah… That is, this failure mode would have been mitigated had the equipment been properly designed.

    “…the CE certification in Europe..”

    There is no such thing as ‘CE certification’. The ‘CE’ mark indicates that the manufacturer has supposedly published a Declaration of Conformity that provides the basis for the presumption of conformity to the scope directives and standards. The ‘CE’ mark does not necessarily indicate that any registered test lab has assessed the product.

  22. I use a Spellman Sl2KW power supply for my main HV supply in my fusion research. Basically 2kw worth of 50kv. An insect made its way into the supply and wound up at a narrow clearance point between the 10kv or so output of the switcher and the Cockroft-Walton voltage multiplier. Due to good design, the supply shut down with all the semiconductors still good, but not before the arc (about double your cheap arc welder’s worth) ate an inch of wire and punched a hole in the (insulated on the inside!) .050″ thick aluminum case. Fairly exciting and stinky. Well, this was a Friday, and I had a dog and pony show demo on the next Monday. Thank you Spellman (CliffS) for helping me get the repair right over the weekend – and for free. I needed to make new wires into a potted transformer – every detail had to be right, re-insulate the re-smoothed Al case and so on. Demo came off just fine. First time I’d had an actual bug (utterly destroyed but it really couldn’t have been anything else) shut me down.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s