iDisplay, webcam multitouch

Embedded above is an interesting multitouch demo by [Lahiru]. The goal of the project was to find an easy way to retrofit current LCDs for multitouch. Instead of using infrared or capacitive recognition, it uses a standard webcam mounted overhead. To calibrate, you draw polygon around the desktop screen as the webcam sees it. The camera can identify the location of markers placed on the screen and their color. iDisplay can also recognize hands making the pinch motion and sends these as touch events via TUIO, so it works with existing touch software. It’s written in C++ using OpenCV for image processing with openFrameworks as the application framework.

[via NUI Group]


  1. Tim says:


    1. Surely it should be able to auto-calibrate?
    2. Not really ‘touch’ is it? It just tracks the coloured pucks.
    3. There’s a pre-written opencv function for tracking colour patches (check out ‘camshiftdemo’).

    Looks like about a day’s work!

  2. ryan says:

    Looks like this video from MS Research, which has been out for a few years:

  3. RussWill says:

    Wow! Thanks Ryan for that new link! It’s nice to go straight to the originator (plus he explains it.) But too bad about lame interviewer (and the lack of annoying music.)

  4. jim slipper says:

    Nice work! I will make you a functional 3d display for your technology.

  5. renter says:

    johnny lee did some amazing work before with a wii controller. I think johnnys idea is more practical especially since you have the option to click with the press of a button.

    However good work though. judos

  6. RussWill says:

    johnny lee is truly amazing with his creativity!

  7. jason g says:

    this is cool its not relly multitouch since u track the pucks but i wanna try this is there a link for a download of the software?

  8. jason g says:

    ps another cool thing ive seen is webcam tracks a laser dot and u use it to move pointer and click

  9. darkblackcorner says:

    Hmm, looks cool to experiment with once the software is released :)

  10. Lahiru says:


    Thanks a lot for the post and thanks a lot for the comments! :)


    >1. Surely it should be able to auto-calibrate?

    Yeah, auto calibration can be implemented.

    >2. Not really ‘touch’ is it? It just tracks the coloured pucks.

    It tracks the circular tangible objects.

    >3. There’s a pre-written opencv function for tracking colour patches (check out ‘camshiftdemo’).

    Well it’s not tracking colours and not using camshift. It is using a custom background subtraction method tht implemented by myself to extract the foreground objects. And after identifying the 2D position it captures the colour info. Since I’m using a normal webcam the displaying image is visible (not like using an IR camera), so removing the background is challenging.

    >Looks like about a day’s work!

    It’s really great if you can do it within a day. :)

    Thanks a lot for the feedback!


  11. poopsniff says:

    iHate this trend….

  12. nubie says:

    So why exactly is he tweaking its nipples in that youtube thumbnail?

  13. TMH says:

    This looks like something that could benefit the disabled with poor motor skills. A cheap and effective way for them to use a computer. Fine tuned of course.

    Great work, none-the-less.

  14. TMH says:

    oh btw. I understand what you were going for, but the music was horrible. lol

  15. Lahiru says:

    hehe yeah! should find a better music track. ;)

  16. roxlu says:

    I saw this post on openframeworks and want to let you know that there I also which I use to write articles on some programming techniques.

  17. Jordan says:

    so…U could exchange the coloured pucks for gloves with colour leds on the tips (activated when U touch the surface)?

  18. Lahiru says:


    Yeah, that’s possible. But I’m trying to allow more natural interactions… using bare hands. :D


    @roxlu: is really cool! :)

  19. uzerzero says:

    i can see some interesting applications for this.. for instance, mounting a small webcam, like the ones that are installed in most laptops, on the bezel by the touchpad. adjusting the angle via software wouldn’t be too bad, just more precise control. this would make for a more portable approach and put us budget hampered folks a step closer to turning our ancient laptops into multitouch ones :)

  20. Jon says:

    Out of curiosity, and I know I am late, would it be possible to have the software calculate a “touch” by watching how much the hand diffuses? Because the ambient light turns the hand into a “blob” to the camera and removes a lot of detail, would it not be possible for it to see the thinning hand/finger/other pointing object and asses when it has made contact with the screen?

  21. Quest12 says:

    Can I use my webcam and my normal LCD screen with this kind of software ? if I can then where I can download this software ?

  22. Webcam Girl says:

    This website needs more comments haha!! Greetz from Denmark

  23. Mitchel Reugt says:

    This is bullshit, copied from MS Research! many users have request the software for try this but no answer, This is called “Open source” right? why are you afraid if you don’t have anything to hide?

  24. Lahiru says:

    Hi Mitchel Reugt,

    Thanks for the comment!

    I need to clean-up the code base before publishing it as a opensource project. it’s my own research and the code is kinda messy. :)

    Anyway since the higher demand. I’ll publish it this month. And please notice that I have nothing to be afraid of. :P


  25. corpucine says:

    Hope u can publish it ASAP.

  26. bowser says:

    Still not published? I would LOVE to use this in my class. I have already implemented a wiimote smartboard, ftir touch screen and home made lcd and doc cam…this would be an amazing add! :) HELP teach our children and POST the software.

  27. Yavol Foxxen says:

    You know, If this went open source, it would probably get really clean, really fast. Not to mention be better, have more options ect…
    Just an idea. – Yav

  28. Shazin says:

    I personally know Lahiru and I have seen the effort he has put into develop this prototype. I am 200% sure that it is his own idea and creation. This is just one of his amazing collection of works. So please just appreciate what you see instead of trying to find bulls.

  29. Lovely cams says:

    I loved this one , I bookmarked your blog!

  30. bowser says:

    Still you haven’t published it :D

  31. Charles Aquilina says:

    Hi, have you published the source and if yes where can I download it.

  32. mac says:

    Never did release that source so I’m guessing that dude had it right?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

Join 96,755 other followers