BiDi Screen, On (and Off) Screen Multitouch


MIT is debuting their latest advancement in technology, a multitouch screen that also functions as a gestural interface. The multitouch aspect is nothing new, the team explains how traditional interfaces using LEDs or camera systems do work, but fail to recognize gestures off-screen.

Gestures are a relatively recent highlight with the introduction of projects like Natal or perspective tracking, but fail to work at closer distances to the screen. MIT has done what seems the impossible by combining and modifying the two to produce the first ever multitouch close proximity gestural display.

And to think, just a couple of months ago the same school was playing with pop-up books.

[via Engadget]

18 thoughts on “BiDi Screen, On (and Off) Screen Multitouch

  1. That looks pretty cool!
    I wouldn’t be able to use it until they eliminated the lag, but cool nonetheless.

    Gotta love the models they used.

    It looks like they’re planning to take this to Siggraph (look at the textures on the wall in the “walk-through” demo).

  2. @RoboGuy
    It seems like the lag would be fixed by actually manufacturing the screen like they suggest in the video, rather than their ‘demo’ screen. The reason I say this is because you currently have lag occurring during the video capture (using those cameras) and the processing of the captured images.If you were to use all built in technology with optimized code it seems like you should be able to make it pretty lag-free.

    We have seen similar technology for a while,
    (notice it measures placement and distance, the same idea would be used on the screen, using different sensors)

    I agree on the models being awesome, and once again I have to hand it to MIT.

  3. I love this project. It’s a great application of coded aperture imaging, basically demonstrating that in many situations where a lensless (pinhole) design is appropriate, it is possible to use a coded aperture to get more light… and depth information as a byproduct.

    Very clever.

  4. This is very cool. I wouldn’t be surprised if high end screens started implementing these features in less than 5 years. In 10, eliminating the need for a mouse or keyboard.

    It kinda reminds me of the screen in Minority Report.

  5. @Poncho-

    Funny… that was my thought too.

    The day televisions come standard with built-in screen-cameras is the day all of my TVs go to the curb. (Particularly if they are fed with an inherently bi-directional conduit, like cable.)


  6. Amazing. A *screen* that *sees*. Wow (not being sarcastic). There are many other applications as well, like videoconferencing where it doesn’t seem like you’re looking at someone else (Assuming an improvement in picture quality). And that joke about someone sticking a sheet of paper to his monitor to “photocopy” it might become a reality :D

    I immediately thought of 1984 as well, but it’s a computer monitor application, not a TV application. TV is not an interactive medium so there’s no point in implementing this. Also laptops all have had an embedded webcam for a while and it seems not to have been abused.

  7. This is very impressive. It sounds simple, but there are a lot of hurdles solved by this project. For one, they’e done a really good job of getting those video cameras working well – after the diffuser there must be very little light hitting the camera, and with the coded aperture, linearity in the cameras pixels is required, something which very few cameras have, and is hard to approximate since sensitivity and noise varies across the scale. Also, with a fine pinhole array like that, the camera must be very high-res, making data processing hard – it wouldn’t surprise me if they weren’t using an fpga even for that demo.

    If this tech becomes sought-after it will push manufacturers to come up with ways to make large ccd’s or similar tech (ie. the full size of the screen), at a still high resolution (the higher the res the more cool features can be implemented, like 3d scanning or webcams from any viewpoint). The amount of data this would generate could be huge (about 500 mb per frame uncompressed, and assuming 30fps, 18 gigabytes per second). Obviously there is a lot of redundancy in this data, and much of it can be eliminated without reducing the devices usefulness. Nevertheless, it still looks like displays would need special purpose hardware to process this data and return a summary to the PC/host device (rather like an optical mouse summarises it’s CCD data to X/Y movement data).

    One very nice benefit of this tech is that the sensors required don’t need to be flawless – using coded apertures a large number of the pixels could be dead, and the device could still obtain good data after a bit of post-processing. This makes the large high res sensor arrays required for this tech far more plausible.

    The output of this could be made “color” by setting each apature to a different colour. Lots of tough postpocessing would be needed to fit together the images though, since perspectives would be different. Alternatively, the sensors could be made R/G/B, but that decreases resolution.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.