Fail of the Week: WS2811 Pixel Failure on FLED


This Fail of the Week project comes from one of Hackaday’s own. [Ben] took on the FLED data visualization project as a way to make the SupplyFrame decor a lot more fun. He had quite a bit of help soldering the 96 WS2811 pixels into their custom made 6’x4′ enclosure and the results are really awesome. In addition to showing server load and playing games, FLED has become something of a job interview. Sit the prospective employee down at a terminal and give them an hour to code the most interesting visualization they are capable of.

But two weeks ago [Ben] staggered into the office and found the display was dead. Did he try turning it off and back on again? Yes, but to no avail. The power supply wasn’t the issue and there was no option but to pull the display off the wall and crack it open for a look at all those pixels. Since every one of them had 4 solder joints on either side he figured the problem was with a broken connection. But not so. He resorted to a binary search for the offending pixel by  cutting the strand in half, and testing each portion. He tracked it down to the pixel whose underside was blackened as you can see above.

[Ben] thinks one of the capacitors inside the sealed enclosure blew, but isn’t certain. Feel free to tell us what you think failed in this component. But the thing we’d really like to know is if there is a more clever way to sniff out the offensive pixel without cutting the connections? Four hours on the floor with this thing (and no knee-pads) and [Ben] has sworn off sourcing pixels from random Chinese suppliers. He might go with pre-assembled strings next time. We chuckle; this is the high-tech equivalent of trying to get old strands of Christmas lights to work.

If you haven’t seen FLED in action, check it out after the break. It amazing how LED intensity and quality diffuser material can make a perfect grid of LEDs seem to dance in waves and color curves.

2013-09-05-Hackaday-Fail-tips-tileFail of the Week is a Hackaday column which runs every Wednesday. Help keep the fun rolling by writing about your past failures and sending us a link to the story — or sending in links to fail write ups you find in your Internet travels.

36 thoughts on “Fail of the Week: WS2811 Pixel Failure on FLED

    1. An interesting idea….not sure I like that one much! I get enough worried looks from our office staff as it is!

  1. >[Ben] has sworn off sourcing pixels from random Chinese suppliers. He might go with pre-assembled strings next time.

    Not sure how buying pre-assembled strings over DIY with discrete ones is going to help with this unless the cause of failure is due to improper handling during your own assembly process. The pre-made ones probably use the same random source(s) from supply chain in China too.

    The only thing that save is the assembly time should(/when???) the next set goes up in smoke.

    1. Thats very true…

      My thought for the future would be to put LED strips in columns, with terminal blocks connecting them, then replacement is as easy as just removing a few screws and gluing in the new strip, makes debugging much easier too.

  2. You could attach it to a lab power supply set with a low current setpoint, and use a current probe to measure the current in either the power or ground lead in various places, using the same binary search pattern.

    If you don’t have access to a current probe (they are quite expensive), or can’t clamp it onto the power or ground wire separately, in case you had used ribbon cable (not the case here), you could also measure either the supply voltage at each LED, or the voltage drop across the power or ground lead. The current in the wires, and therefor the voltage drop across them, will be much larger between the short circuit and the power supply, while the current will be very close to nothing when you have passed to short circuit.

    This would seem rather trivial, actually; I’m a bit disappointed [Ben] or any of his coworkers couldn’t figure this out.

    1. Well the main problem is we don’t have a lab power supply that can do this. If we have the appropriate tools then yeah this gets a little easier, but without a decent powersupply you can’t actually energize the circuit and debug it actively. This is after all a software development office not a proper lab.

      Like the suggestions though, soon as we get the HackASpace setup we’ll be kitting it out with appropriate equipment so this sort of debugging will be easier.

      1. You could improvise a current-limited power supply with a resistor. Or even an appropriate-voltage lightbulb. Connect a 5v (6v should do) lightbulb in series with the 5v power supply, it will light up. The current will flow through the shorted LED, and may produce a limited amount of heat (which would allow finding it by touch) and wil produce a votage drop along the wires which can be sensed with a millivoltmeter. (A sharp pin can serve as a probe for piercing the wire insulation.)

        A lightbulb in series with the circuit can also serve as a protection against short circuits. Especially handy for mains voltage and low-power loads (e.g. for testing or repairing small switching power supplies).

  3. Was there any evidence of this blowout from the front? That’s my first debugging step: Does anything look crispy?

    Followed by the three other relevant senses:
    Does anything smell crispy? (if so, can you tell where the smell is strongest?). Does anything feel warm? Is a specific part making unusual noises?

    I haven’t found a good use for taste yet.

    1. Unfortunately we just had a general burnt smell that actually I traced down with my schnozzle to some of the wire insulation melting further along the chain, actually had to replace a couple of wires that had melted through their insulation.

      The broken LED in question didn’t smell at all, I suspect everything had remained encased in the silicone.

      1. Also, out of curiosity, why did you daisy-chain instead of bus the power? Metal strip with threaded holes, do row of LEDS, VDD, row of LEDS, GND, etc. Then you only daisy-chain the DIN and DOUT lines, which would probably simplify debugging by a fair amount (you could also then put a fuse on each individual VDD rail, when one LED blows and shorts itself, it blows a fuse and you have a smaller pool of potential baddies to worry about)

        1. Thats interesting…

          Honestly we didn’t really think through on that one, we just decided to recreate the LED strip that the LEDs had originally came in. We did end up having to add a few additional power taps along the strip to account for voltage drop, bus power would have helped there too.

          Thanks for the idea!

        2. This. Most Chinese trips have so much voltage drop that one end is noticeably darker than the other. With these, I’d be worried about the current in the strip adding to the internal dissipation of the parts.

  4. The magnetometer inside a phone is sensitive enough to be able to trace up to where the high current is going.
    I guess it is important also to limit the current, so a bench supply or just a power resistor should be good enough.

    1. Now thats the sort of hack we could have done!

      Power resistor we could have pulled from our busted replicator 1’s hotends. I had no idea the magnetometer was sensitive enough for that, genius.

  5. i use these all the time at work. when we have a pixel die and we need to replace it we power the system ON and look for the 1st pixel that is OFF in the chain. once the 1st OFF pixel is found we check it and the pixel just before it because more often than not the pixel just before the OFF one is the bad pixel. we replace the pixel and test the system. the whole process takes us about 15-30min

    1. That’s not what happens with these LEDs in this particular failure mode. Because there’s a short interfering with both the data line as well as the power lines no LEDs light at all.

    1. Its about 6″ apart or so, whole board is 6ft x 4ft. The LED modules are pretty beefy, having 3 RGB leds in each unit about an 1.5″ in diameter. They were the modules I had leftover from the Diodome project for last years Burning Man.

  6. Sometimes you can find this kind of thing with a thermal camera.
    (Of course, you have to have one of those.)

    1. If you don’t have a thermal camera, a fine mist of acetone, alcohol, or some other substance that boils off at room temperature can point out the hot parts. I.e. the part that dries faster than the rest is probably toast.

      1. I use this method on engines to quicly determine what cylinder isnt firing on a cold start , mist brake cleaner over the exhaust and its very obvious which cylinder isnt running. Handy on diesels. Never thought of using it on electronics. Brake cleaner is AFAIK the same as pcb cleaner.

  7. Us christmass light guys are very used to this kinda thing unfortunately its very common, with randome pixels, from china. Even the latest thing that has been happenign is voltage regulates in the pixles were popping on non 5v strings.

    Generally debugging, is find the pixel having issues and replace it and the one directly behind it in the chain often it is the pixel right before the problem.

    Another kinda solution if it was the regulator the went pop is run the pixel at 8-9v instead of 12 so it is dissipating less heat, once again if these are 12v ones, if there 5v i don’t know, because power supply issues generally take out the first few not one halfway down the line.

    1. Also if you want nice quality ones google Pixel Bulbs there were done by one of the Christmas light forum users @ i don’t think there in stock at the moment though.

    2. Hmm undervolting them might be a good idea, as long as the brightness drop off isn’t too large (as this is in a well lit office environment right now). That would at least extend the life of them anyway. Thanks for the suggestion.

  8. I built a WS2812 array out of 60/m strips, I was shocked at how frequently they’d fail. During development even the smallest of physical bending, or possible small PSU surges or fluctuations (as far as I could work out) would cause a row blackout.
    Oddly, invariably the first or second one in a chain. You’ve really got to secure them down to something as they’re just not happy free roaming around breadboards and arduino pins.
    As far as I’m concerned they’re pretty poorly QC’d, and may even have inherent design flaws. If an led fails in a chain it’ll totally fail to transmit / smooth out the signal.
    Failure rates for my project was probably over 10%, and I treated them really carefuly.
    They’re like the modern equivalent of old Christmas tree lights. Even the signalling protocol sucks.
    I’m waiting for the next addressable RGB LED ‘standard’ to come out of China that has been a lot better thought out.

    1. You will be waiting forever, because the WS28XX line is NOT well though out at all. It’s pretty much a dirty hack.

      Next from china will be another dirty hack. we will never really get a good clean and reliable anything out of china at the $0.59 per pixel price point I am buying WS2812’s for.

  9. Also, since the display power supply didn’t fry or detonate the pixel when the pixel failed, it had good short circuit protection. Given that, it would probably have been safe to use the normal power supply as an excitation supply when probing for the short. I.e. you had the system fully powered for several minutes while confirming it was dead, 5-10 minutes more while probing for the problem shouldn’t do any more damage.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s