Converting Live 2D Video To 3D

Here’s some good news for all the fools who thought 3D TV was going to be the next big thing back in 2013. Researchers at MIT have developed a system that converts 2D video into 3D. The resulting 2D video can be played on an Oculus Rift, a Google Cardboard, or even that 3D TV sitting in the living room.

Right now, the system only works on 2D broadcasts of football, but this is merely a product of how the researchers solved this problem. The problem was first approached by looking at screencaps of the game FIFA 13. Using an analysis tool called PIX, the researchers both stored the display data and extracted the corresponding 3D map of the pitch, players, ball, and stadium. To generate 3D video of a 2D football broadcast, the system then looks at every frame of the 2D broadcast and searches for a 3D dataset that corresponds to the action on the field. This depth information is then added to the video feed, producing a 3D broadcast using only traditional 2D cameras.

Grab your red and blue filter shades and check out the product of their research below.

32 thoughts on “Converting Live 2D Video To 3D

  1. Almost any 2d video can be given a serious 3d feel simply by playing side-by-side a second copy of the video that is delayed by a fraction of a second. As long as the action isn’t too fast the effect is remarkable.

    1. The BBC released cardboard “3D” glasses for a system a bit like that a few years ago. The lenses were vaguely purple and yellow, but the main difference is one was darker. Darker images take a little longer to appear in the eye, so there’s a delay. Certain videos will give the appearance of 3D. In this case, it was rotation, lots of sequences where the camera spun around things. One eye sees the image from a little while ago, creating the difference, and it works. Pretty well, too. Although the need for rotation made it a definite novelty, it would become annoying pretty quickly in large doses.

      1. I’m pretty sure that was ColorCode 3-D. It uses amber/dark blue lenses and works the same way as the red/cyan anaglyph but has much better colour reproduction and much less double imaging when not wearing the glasses.

        1. No, the colour of the TV picture was normal during the 3D sequences, exactly the same as during ordinary broadcast. And every “3D” sequence involved something rotating. I remember cos obviously being a geek you take the glasses off partway through to try figure out how it works. I think it was a Children In Need thing. Glasses were distributed in supermarkets.

      2. Channel Four did the most recent 3D anaglyph cardboard glasses demo in the UK, with Blue and Amber glasses in around 2009. They showed the 1953 Coronation (which was shot in full colour 3D) and a few other things. https://www.youtube.com/watch?v=BWFlClq0Rj8

        The BBC used a different system for Children in Need in 1993 that used the Pulfrich effect, and didn’t required any 3D camera, it just used one eye with some ND filter in it, which made all tracking camera moves appear 3D. It’s a mental trick rather than any real 3D content. Can’t remember if the glasses were a purchase that generated money for the Children in Need charity or whether they were free with the Radio Times. The Pet Shop Boys performance was pretty good (the camera never stopped moving) https://www.youtube.com/watch?v=mx-OwFxOFjs

        TVS (the ITV franchise holder in the South of England from 1982-1992(, in the UK, did the anaglyph red/green (or was it red/cyan?) thing in 1982 with a test show as part of their “The Real World” science series, and also showed a few 3D films around the same time. Was quite fun at the time. Think the glasses came with the TV Times. https://www.youtube.com/watch?v=mma_lTw3_EA

        1. Also I remember in the Museum of Photography, Film, and Television (now called something else, more TV-based), seeing the Coronation film. They had 2 versions, one for red-green, one using polarised light glasses. Obviously you can’t use the polarised method over TV, they must have set up a special projector.

    1. works well as in fakes it badly? yes.

      The problem is there is ZERO reason for a video game to not support 3d. it is effortless for all game consoles to have every game in 3d. And honestly THAT is where 3D tv’s would have shined and really had a huge following. Forza 5 would have ROCKED big time to be in 3D.

      1. My 2d-to-3d works fine actually, and personally I don’t use 3D in games.

        But rendering from view points seemed to slow the Call of Duty I had on the PS3 to the point that it lagged so it’s hardly zero effort…

        I use my TVs 3D feature to allow full screen split screens using dual play glasses…

      2. Yes, there is one very significant reason to not have 3D by default: It takes twice the rendering power. For games that can already run at 120FPS or higher, yes, it works, but for those pushing the hardware to its limits to just manage 40-60, they become too choppy to be tolerable when cut in half for 3D.

      3. It’s not effortless to convert a game to stereoscopic 3D, there are a lot of things that won’t work out of the box.

        Basically all HUDs, GUI elements, billboards, crosshairs, skyboxes, particles, shaders and shadows that are not rendered at the correct depth or effects like relief mapping that look bad in S3D. The x2 performance hit doesn’t help either and requires heavy compromises on the rendering quality.

        That’s why most games not developed with S3D in mind don’t work well with the NVIDIA 3D Vision driver. There are numerous publications from NVIDIA and Sony that explain how to implement a game to support S3D correctly. This one for example : http://www.nvidia.com/content/gtc-2010/pdfs/2010_gtc2010.pdf

  2. Nothing but sports? At 50 feet all static 3D is nil, nil. It’s motion that does the trick. Lookup 3D with one eye, that works too on your screen!
    And yes about the delay with two screens. At a sports bar here they have four in a row behind the bar. Two had the same game on, one with captions enabled which delays the video I guess. In slow pans the effect was good. I use the cross-eye mode which needs the left-right reversed, but no glasses or devices.

  3. I don’t know is 3D Tv still a thing? for a few years I thought it was going to happen there but now I think we are heading in the direction of just 2D. Correct me if you think I am wrong.

        1. The only time I ever use the 2d-3d conversion on my set is if I really want to watch something in 3d that was never released in 3d – like the Rocky Horror Picture show this Halloween. 2d-3d, while good, is more of a gimmick though. I miss the old 3d movies that actually took advantage of the “pop-out” effects. Most 3d movies today focus more on “depth” than effects. Meh… I’ve been buying everything 3d I could find since I got my first field-sequential converter. None of that red-blue crap though. It looks amazing on still photos, but looks awful for videos.

    1. It’s a bit annoying how 3D is always everything or nothing, and you have to wonder why they can’t get that there is just a small percentage of people that are into it, and so you could make a small percentage of programs available instead of trying to make everybody like it or completely abandoning it.

      And as you said, after they got all the equipment to do it too, what do they do with it all I wonder, all those 3D cameras for instance, and all the investment.

  4. Inverted stereoscopic imaging with a calibrated camera. *YAWN* Throw enough processing power at a LUT and remap the pixels based on their position and you could even watch the game in 3-D with a God’s Eye view. Interesting to people who don’t understand OpenCV. But at least someone is out there trying to entertain those yuppie folks who spent tons of money on 3-D televisions.

  5. Out of all the sample given in the article in the comments only the field hockey came close to what I’d expect to call 3D, in some portions, but even it sucked in general. I couldn’t see myself watching an entire game.

    1. Really? The soccer looked pretty 3D to me (using side-by-side to view it).
      But I noticed it made me very aware of the wide-angle lens used and when you swing around wide-angle lenses some disorientation will occur, depending on your sensitivity to it. that could be bad.

  6. I just don’t see 3d vr tv ever catching on. The reason so few DVDs employ the multi angle view specification is that cinematographers have learned to rely on FRAMING to the exclusion of everything else. When you can see that the victim must certainly be able to see the monster from his perspective you don’t get the shock of the monster “suddenly” appearing. Producers don’t want to be forced into the real world with all it’s difficulty of choreographing an adequate suspension of disbelief. Video game producers were raised on this mode.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.