David Krum: The Revolution in Virtual Reality

[David Krum] is associate lab director at the Mixed Reality Lab at the Institute for Creative Technologies at USC. That puts him at the intersection of science and engineering: building cool virtual reality (VR) devices, and using science to figure out what works and what doesn’t. He’s been doing VR since 1998, so he’s seen many cool ideas come and go. His lab was at the center of the modern virtual reality explosion. Come watch his talk and see why!

What is Virtual Reality, Really?

We all know what VR is, right? It’s that trope in sci-fi where you’re stuck in the Matrix, unable to tell that you’re not actually in the real world. All of your senses are simulated so well that you’re fooled into generating power for the hive mind. [David] has slightly more realistic goals.

vr1For [David], it’s about immersion. And the gold standard for him is that immersive VR makes you act as if the VR world were real. It’s not that you can’t tell the difference between the real and the virtual, but that you behave in and react to the VR world in the same ways that you would to the real. If you’re engaged, the VR experience is a success.

If the goal is a VR world that helps train military or pilots to operate their machines and work in teams, does it matter that they can’t actually tell what the inside of a tank smells like? Instead, what’s important is that the “sensorimotor contingencies” are right — that what you’re doing affects the world in the right ways. In the earliest Oculus Rift headsets, for instance, they only tracked motion with a gyro, so you could look around the world as if you were in a fishbowl. But when you strafed (moved your head side to side) it wouldn’t react. Later versions fixed that, and became significantly more immersive.

vr4But VR is not all about the hardware. How the scenarios are crafted and even the quality of the art matter a lot. In one of the more poetic moments of the talk, [David] insisted that you should be able to leave footprints in the sand. He was talking about how the user changes the virtual world, but he’s equally interested in how the VR world changes the user.

In all of these metrics, VR has made tremendous progress in the last decade, and [David] thinks it’s going to continue. What made this all possible? Affordable VR headsets. When a VR headset cost $60,000 per unit, no matter how immersive the experience was, nobody was going to use it for teaching children, or as therapy for stroke patients. With cell-phone-based units as cheap as $10, the landscape is dramatically different.

For [David], the Oculus Rift was the breakthrough device, because it demonstrated that there was market demand for a “cheap” VR device and encouraged other companies to make competing products.

The Myth(ology) of the Garage

So where did the Oculus Rift come from? If you believe their marketing hype, it was created by a genius gamer kid alone in his parents’ basement, who overcame all of the stumbling blocks that had plagued VR for decades. This plays into the Silicon Valley mythos of the company that starts off in the garage and goes on to change the world, and [David] gives the story a refreshing debunking.

vr3HP “started” out of a garage. Well, kinda. The early prototyping and R&D actually took place in a lab at Stanford University. Apple started in Steve Jobs’ garage, right? There was no design or manufacture done there, but they did use it for testing and temporary warehousing before shipping computers to stores. Most of the conceptual work was done in a space at the Stanford Linear Accelerator, and of course in the community of the Homebrew Computer Club. Google had a garage — that they rented out after they had a few million dollars in seed investment. But of course, the initial work on Page Rank was already done by then. And when was that work undertaken? While Sergey and Larry were graduate students at Stanford.

The point is, the Silicon Valley garage mythos is a myth. The idea of the special whiz kids working along in their garage or basement just doesn’t stand up. In all of these cases, the founders were actually part of a larger community that was working on the problem. And in all of these cases, the community was also an academic one, but there’s no reason to think that the next big ideas won’t be coming out of well-equipped hackerspaces or anywhere that there’s a “community and a culture that creates and shares ideas”. (We’ll take that as an unintended compliment!)

vr2[David]’s lab has been interested in making low-cost VR a possibility since 2010. In 2012, for instance, they built a cardboard VR device with lenses that you could slip your cell phone into. They were able to make them so inexpensively, using a laser cutter, that they gave them out for free at a conference. In 2014, we’d see this idea hit the mainstream as the Google Cardboard.

They’d been doing smartphone-based VR work since 2011. Some people at Samsung actually wrote them an e-mail thanking them for open-sourcing their work and inspiring the development path for the Samsung Gear VR headset. And, of course, [David]’s lab hired and trained a young kid as a technician, who would later go on to found Oculus VR. “In his garage.”

What’s Next?

vr5Now that VR headsets are ubiquitous and cheap, what’s next on the horizon? It’s no secret that VR is going to be dominated by gaming for the next decade. But [David] has hopes for other uses as well: using VR to make us better artists, engineers, builders, and makers. VR will also have a role in education, as people can take tours and interact with machines that they never could in real life.

[David]’s fear about the future of VR is that, as the major gaming companies fight to segment up the industry, it will lead to walled gardens. If you want to play Sony’s games, you need to buy their headset, pay for the games in their store, and essentially confine yourself to their world — and contributing to their bottom line.
vr6But if we can avoid falling into that trap, [David] sees a lot of new and interesting opportunities for VR. You know how sometimes you have to walk back downstairs to remember what you were thinking about when you were in the kitchen? That “context-dependent memory” is an interesting area of VR research. There’s a lot to learn about people in VR: how people interact with virtual characters, and what that says about our ideas of personal space. And there are opportunities for therapy. [David]’s lab does work with stroke victims, testing if moving their arms in a VR world can help them to regain control over their own real limbs.

vr7Then there are open avenues in making VR itself better. [David]’s lab did some especially neat work integrating stop-motion animation into a VR context. It turns out that the real-world textures, lighting, and other cues can help make objects seem more real than just a collage of polygons would. And one of the coolest applications was actually an extension of the cardboard viewer: a 3D printed overlay for a tablet computer that allows the top third of the screen to function as a shareable VR platform, while the lower part is visible by everyone in a small group. This “casual VR”, where people can pass the tablet around without having to strap on invasive headsets, promises to make the technology a much better fit for ad-hoc group collaboration.

The Final Lesson

The new boom in VR didn’t come out of nowhere — it came out of work done across research labs like [David]’s, working in the open and sharing their innovations. These are folks who’ve been working on VR since the 1960’s, and the ideas behind the new crop of headsets are nearly that old. The technology has caught up, with higher resolution screens and better graphic processing, of course. But what really drove companies to invest, compete with each other, and eventually drive the revolution forward, is that prototypes were built that demonstrated the possibilities of a less-expensive VR experience. How’s that for a moral to warm the hacker’s heart?

18 thoughts on “David Krum: The Revolution in Virtual Reality

  1. > The idea of the special whiz kids working along in their garage or basement just doesn’t stand up

    Seriously, get your facts straight, the whole history of the Rift is public on the MTBS3D forums since 2009. Palmer Luckey was hired by Mark Bolas to work at the Mixed Reality Lab because he’d built a prototype of a VR headset by himself. You can follow his progress along the years on the forums. Also the cardboard headset from the lab (FOV2GO) was designed by him while he was there as well.

    For more details see here : https://www.reddit.com/r/oculus/comments/5312c9/oculus_cofounder_palmer_luckey_screaming_in_a/d7pxk8a/

      1. Sure, it dates back from the 60s, but it has failed to gain any success in the consumer market since the VR boom of the mid 90s. It’s only since the apparition of a low cost, wide FOV headset in 2012 that it has started to take off.

        Concerning your link, John Carmack has already said that Oculus uses zero line of code that he wrote while he was at Zenimax, he acknowledged that the Rift prototype was from Palmer Luckey (he even posted a review on MTBS3D) and during the past 4 years ZeniMax has not produced anything related to VR.

        Anyway, the trail begins today, we will be fixed in 3 weeks.

      1. I read your comment, but I did not find an answer to my question.
        What I already know is that the PSVR display does indeed run natively on the HMDI output of a graphics card (without the PSVR box in between the HDMI connection). However, so far, I was unable to achieve information about the supported modes (such as 60Hz support?) of the PSVR display (again, without the box).

        Regards, Lars

        1. That’s exactly what the quote I posted said. The full text before it was :

          On stage, Norden clarified the functionality of the breakout box as part of his presentation:

          “What does it do?”

          “- Object-based 3D audio processing”
          “- Displays social screen”
          “- Mirroring mode”
          “- Separate mode”
          “- Displays PS4 system software interface in “cinematic mode””

          “Norden was adamant about getting the point across because he says it’s been consistently misreported by some gaming and tech press that the box is assisting the PlayStation 4 with VR rendering.”

          So it’s pretty clear, the display and GPU both support 120 Hz, without the help of the breakout box.

          The official specs are listed here, it also supports 90 Hz : https://www.sony.com/en_us/SCA/company-news/press-releases/sony-computer-entertainment-america-inc/2016/playstationvr-launches-october-2016-available-glob.html

          Now it’s up to the games to support 120 Hz, 90 Hz or 60 Hz with reprojection to minimize performance impact.

          1. Yes. But it is my understanding that the box does reprojection on a 60Hz, so that a 120 Hz signal is passed to the PSVR display. A 90Hz input to the box is passed as a 90Hz signal to the PSVR display. The sony URL that you posted, states: “Refresh rate 120Hz, 90Hz”.

            My question is: Without using the box for the HDMI signal, does the PSVR display support 60 Hz? I know that it supports 120Hz. I know that it supports 90Hz. But does it support 60Hz for sure?

            Regards, Lars

  2. @Lars R. No, it’s not the breakout box that does reprojection, it’s done in software just like for the Vive and the Rift. It’s explained in this article : http://www.roadtovr.com/sony-confirms-new-90hz-display-mode-for-playstation-vr-formerly-morpheus/

    “Sony went on to explain that two display modes would be available: one where the game would render at 60Hz and be bumped up to 120Hz for display using an using an interpolation method called ‘asynchronous reprojection’. The other mode would allow developers—who wanted to take advantage of it—to render and display natively at 120Hz.”

    It doesn’t need to specifically support 60 Hz since sending two identical frames successively at 120 Hz is basically equivalent to having a 60 Hz display. The added benefit apart from the performance gain is that you’re still refreshing at 120 Hz, so you don’t have flicker like on the Gear VR for example.

    1. This is news to me. Do the GPU of the Playstation does the reproejction itself, after the game rendered at 60Hz, right?

      However, I only have a pixel rate of 165MHz and thus, I can’t provide 90Hz. (1920*1020*90=176MHz). I can do 60Hz, maybe 75Hz.

      On reddit someone mentioned that he connected the PSVR to the Raspberry PI without using the cinema mode (and I think, even without using the box). Could you imagine, which configuration and which mode he used?

      Regards, Lars

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s