There’s a small but dedicated group of folks out there who spend all year planning their Christmas decorations. These aren’t simple lawn ornaments or displays, either, but have evolved into complex lightning performances that require quite a bit of computer control. For some things, hooking up a relay to a microcontroller can get the job done, but [Andy] has turned to computer vision to solve some of the more time-consuming aspects of these displays.
Specifically, [Andy] has a long string of programmable RGB LED lights to wrap around a Christmas tree, but didn’t want to spend time manually mapping out each light’s location. So he used OpenCV to register the locations of the LEDs from three different camera angles, and then used a Python script to calculate their position in the 3D space. This means that he will easily be able to take the LEDs down at the end of the holidays and string them back up next year without having to do the tedious manual mapping ever again.
While [Andy] notes that he may have spent more time writing the software to map out the LEDs than manually doing it himself, but year-after-year it may save him a lot of time and effort, not to mention the benefits of a challenge like writing this software in the first place. If you want to get started on your own display this year, all you really need is some lights and a MIDI controller.
I’ve always wanted to do this exact thing with my IOT christmas tree, though I didn’t have quite the same density of lights (along with other issues) to do anything meaningful with it. This is so cool! I think it’d be great to project GIFs on to.
Also according to the readme, it sounds like this is meant for multipurpose festive foliage, not just christmas but halloween too.
This SO exactly fits the meme of neighbors competing for the most awesome Christmas display, but I watched the video anyway, to see why it was so important to map each LED’s position in space. That is pretty damn nice. Good luck getting each light in the same place next year..
He won’t have to put it in the same place next year – that’s the point. Just has to auto-map it again. This process would work for any structure/installation the lights are attached to – whether its a different structure or the same structure but a different “wrapping” the next year
Thats kind of the point. System should make it just as easy next years lightup. No grid or banding system for the pixels individually. Not exact pisition dependent. Given that trees grow and shift is probably good thing. Put strips up. Set NeoPixel. Take pictures. Determine and plot position from picts. Manipulate array to match whatsoever patterns. So much lazier.
It would be easier if each NeopPixel had BT-5 prox detection. Maybe next gen NeoPixel.
You don’t even need to be able to map every LED as long as you can detect enough of them. You can expect that the missing LEDs are between the detected LED’s and if you’re being fancy do some curve fitting but otherwise assume a straight line. That’s what it seems the Twinkly brand of addressable Christmas lights do. You can map them in 2d or 3d with your phone’s camera.
I don’t mean to say that what he has done isn’t super cool or an achievement. Because it is. This is an awesome project and I’m glad he shared it.
The problem with doing that is these are string lights, not LED strips, and each LED is spaced out from the previous by several inches, and each LED could be pointing a different direction further skewing the calculations, so if accuracy matters at all, then inferring the LEDs in between isn’t such a good method. I know, I’ve done it before.
Cool, you have real world experience (starting years ago!).
How did you handle partially obscured LEDs in strings? Just take enough images from enough angles that you can calcuate all LED locations?
Do you have any code to share?
I actually just added cubic interpolation to the code this evening to fill in the pixels that I missed during the initial mapping.
Oh that is cool. Thanks for the update.
This is the killer feature I’ve been looking forward to with diy addressable LED projects. Since I don’t have the skills to do it myself.
I’m definately going to take a gander at your GitHub repository.
BTW, the (commercial) Twinkly products do this in their app. It’s really quite interesting to watch it sequence the lights to detect each position.
If anyone knows which LEDs the Twinkly series 2 uses – they seem to be individually addressable but I don’t think they’re WS2818’s – I’d love to know. As much as disassembling it is tempting, the $400 price tag makes that a little risky. I am 99.9% sure they’re driven off an ESP32.
Disclaimer: not associated with the Twinkly folks. I just impulse purchased their early product on sale a couple of years ago, and I really love them.
First generation was using ESP8266 (ESP-WROOM-02 module) with firmware freely readable from flash memory (and also included in apps).
Newest Gen II use ESP32 (ESP32-WROOM-32D), with firmware protected by AES-256 encryption (both in flash and in files). And FLASH_CRYPT_CNT is already at its maximum 0x7F value, meaning encryption can’t be disabled anymore, and that you can’t flash any unencrypted firmware using serial interface. Don’t know if it is a intentional to forbid any modification, or if it a result of multiple in factory flashes for whatever reason.
LEDs are very special ones, since they use only 2 wires providing both power supply and driving signal. Power supply is 24V.
They have several models: classic RGB ones, but also RGBW ones.
I have alread asked here a few times if anybody knows which ones these leds are, but never got any answer yet.
I haven’t investigate what protocol is used.
Gen II also include a microphone, with a function to enable sound reacting animation. Doesn’t work so well because mic is inside a completely sealed water-tight enclosure. But they have a new product called “Twinkly music”, which is a kind of USB dongle, with IA supposed to react to music and drive leds animation accordingly.
Wifi protocol has been reversed: https://github.com/scrool/xled
F-Secure labs have investigated this protocol 2 years ago, and found it quite insecure.
https://labs.f-secure.com/blog/twinkly-twinkly-little-star/
Don’t know if this has been improved since then.
App with computer vision mapping works quite well, but last year i had some bugs: a few leds were mapped at wrong position, so were displaying colors noticeably out of sync with the expected animation. I haven’t tried yet this year if these bugs are gone with all the updates they made to both app and firmware.
You can also combine several strings to work together as a group, extending your animation to all the leds, or synchronizing different strings together.
One frustrating limitation was that only 1 animation at a time could be configured to run in free run/untethered mode. But in latest app update a playlist mode is now available (not tested yet).
You can also control them in real time through Wifi (and bluetooth for Gen II), but since only phone apps are provided, it is quite limitating. Much more could be done with some computer (Raspberry or desktop/laptop).
They also have “pro” products, with Ethernet connected controllers to drive a lot of leds.
Overall they have very great products, but i believe that a lot more could be done by hacking them!
Twinkly products are made by Ledworks, which is an italian company:
https://twinkly.com
https://ledworks.io/
I ended up dealing with the Twinkly folks when I had three sets of their gen1 strings brick themselves during a firmware upgrade. They immediately sent me the (far better) gen2 model with 400 strings, and a prepaid shipping detail to get the failed units back to them. I couldn’t fault that for customer service. They also added me to their beta program for the Twinkly Music.
Thank you for the above links. I too look forward to trying their new features (unpacking the tree today).
This is purely a guess, but I suspect their focus on security is to prevent knock-offs. Although they denied it, I did wonder if they suspected my bricked units were counterfeit. From the tear down, it would seem to be a readily replicatable piece of hardware.
I did a little patent searching, and their position mapping technology is in WO 2019/087165 Al, assigned to LEDWORKS SRL. Most of their patents seem to be from “Marco FRANCIOSA”.
LEDWORKS SRL has three patents:
Lighting system and method for controlling the switching on of lights emitted …
WO EP US IT US20200296814A1 Marco Franciosa Ledworks Srl
Priority 2017-09-18 • Filed 2018-09-18 • Published 2020-09-17
A lighting system includes a plurality of lighting members and a power source for those lighting members. The power source has of a plate-like member connected to an electrical power generation unit and a plurality of housing seats for removably attaching one or more lighting members, the housing …
Optoelectronic device of the led (light emitting diode) type or the like
WO WO2020141450A1 Marco FRANCIOSA Ledworks Srl
Priority 2019-01-03 • Filed 2019-12-31 • Published 2020-07-09
Optoelectronic device of the LED (light emitting diode) type or the like, comprising at least two emitters (20,21,22), the at least two emitters are contained within an outer transparent envelope (12), so as to allow the passage of the light generated by said at least two emitters. Each emitter is …
Method and system for disposing a plurality of lighting devices
WO WO2019087165A1 Marco FRANCIOSA Ledworks Srl
Priority 2017-11-06 • Filed 2018-11-06 • Published 2019-05-09
Method of arranging a plurality of lighting fixtures (1a-1q) in a room or on an object (4). The method comprises the steps of : a) producing a virtual model (41) of the room or the object (4) in or on which said lighting fixtures (1a-1q) are to be placed, b) virtually arranging said lighting …
WO 2020/141450 A1 is probably the most interesting, as it seems to suggest that they’re actually doing LED development themselves.
US 2020/0296814 A1 describes a fairly generic individually addressible LED system, which doesn’t seem particularly interesting.
Marco himself has a couple more (pre-Twinkly), but they seem to be early iterations of the idea.
I don’t know what to make of this. Hopefully, someone can tear down one of these LEDs and get a die shot. My mind goes to the idea, maybe, that they’re using something like a Padauk PMS150C (“3c microcontroller”) in each LED. Their volumes surely aren’t high enough to justify a custom ASIC, but that would work well. The idea of my LED string having 400 microcontrollers is DEEPLY appealing to me!
I don’t know how you had good customer service my experience was horrible I lost out on a bunch of money and they don’t seem to take ownership for flaws in there own products and the generation 2 if you notice come with ZERO PRODUCT WARRANTY so anyone looking into twinkly there lights are great when they work right and as far as customer service goes good luck because it’s a roll of the dice consider them the house you have already bought the product so the house wins no matter what if you haven’t bought any I suggest really checking them out and if you receive them with any issues what so ever take them back to the store and have them replaced YOU DO NOT WANNA DEAL WITH “customer service”
You seem quite knowledgeable. Question for you. I accidentally cut through my strand of rgbw lights when trimming a zip tie. I cut through all 3 wires. One clearly runs the entire length and an easy solder. However the other two are more difficult to determine which should be connected. I’ve tried to look up patent and wiring diagrams but am unable to find anything. I’m also not sure how to test the wires to ensure I connect them properly. My entire strand is out when I plug them in. (250led rgbw) which splits after the WiFi box. It’s a midline cut on one side. None work. Any thoughts on how to fix without damaging them further?
I cut all my twinkly strings to build in an extension up a large tree. Once the extension was added. Worked and maps fine. I did this with 7 strands (14) extensions. One of which I reversed twoindividual wires in a strand. I reversed back and all worked fine. I also had a rat nibbled right Through a link. This caused a weird effect. I spent a long time tracing lines. Once I connect back what looked correct, everything works and maps – for about 2 min then six led’s, near the cut, go haywire. I figured the string was spent, so I tried different combinations of links. Accidentally found that if I leave one link disconnected, it all works fine. No idea why but all the combos didn’t fry the line and I have left the one disconnected – in the ”if it is not broken, don’t try to fix it” line of thinking. It has been link that and working for about 5 months now.
I found a YouTube link of someone explaining how twinkly lights works and what type of led they use
https://youtu.be/QvBOAuIe4cs?si=s3xVOXlPctjhZx2F
Awesome! cannot wait to see the projector version… So no strings have to be installed but yet the lights look installed correctly on the house features with the same effects.
Next up, use the 3D position data to determine how the LEDs are switched. Draw a plain through the model and rotate or warp it, any LEDs that intersect it are changed.
It might be interesting to use this technique as motion capture for wind movement or even make some permanent mounting points and take a time laps of the growth over several years.
Not to worry, let somebody else search this for you on your own site:
https://hackaday.com/2017/12/21/bringing-a-christmas-lights-show-inside/
https://hackaday.com/2015/12/30/must-have-overkill-christmas-tree-lights/
Indeed, those are two very similar projects.
The freeware xLights program has this feature as well. https://manual.xlights.org/xlights/chapters/chapter-five-menus/tools/generate-custom-model
Why aren’t we talking about this? XLights has had this for awhile plus the layout and sequencing with no effort on your part.
The xLights function works for 2D models where all pixels are visible from a single camera. This is a similar function but for 3D models where not all pixels are visible from each camera location.
I did this about 5 years ago, using 4 cameras to make the calculations more accurate, because LEDs do get occluded by the tree and I wanted all 4 sides accurately mapped in 3 dimensions. Kudos for someone finally catching up to me, lol. Programming interesting visuals to display on a 3d irregular matrix is another bit of fun.