Virtual reality has seen enormous progress in the past few years. Given its recent surges in development, it may come as a bit of a surprise to learn that the ideas underpinning what we now call VR were laid way back in the 60s. Not all of the imagined possibilities have come to pass, but we’ve learned plenty about what is (and isn’t) important for a compelling VR experience, and gained insights as to what might happen next.
If virtual reality’s best ideas came from the 60s, what were they, and how did they turn out?
Interaction and Simulation
First, I want to briefly cover two important precursors to what we think of as VR: interaction and simulation. Prior to the 1960s, state of the art examples for both were the Link Trainer and Sensorama.
The Link Trainer was an early kind of flight simulator, and its goal was to deliver realistic instrumentation and force feedback on aircraft flight controls. This allowed a student to safely gain an understanding of different flying conditions, despite not actually experiencing them. The Link Trainer did not simulate any other part of the flying experience, but its success showed how feedback and interactivity — even if artificial and limited in nature — could allow a person to gain a “feel” for forces that were not actually present.
Sensorama was a specialized pod that played short films in stereoscopic 3D while synchronized to fans, odor emitters, a motorized chair, and stereo sound. It was a serious effort at engaging a user’s senses in a way intended to simulate an environment. But being a pre-recorded experience, it was passive in nature, with no interactive elements.
Combining interaction with simulation effectively had to wait until the 60s, when the digital revolution and computers provided the right tools.
The Ultimate Display
In 1965 Ivan Sutherland, a computer scientist, authored an essay entitled The Ultimate Display (PDF) in which he laid out ideas far beyond what was possible with the technology of the time. One might expect The Ultimate Display to be a long document. It is not. It is barely two pages, and most of the first page is musings on burgeoning interactive computer input methods of the 60s.
The second part is where it gets interesting, as Sutherland shares the future he sees for computer-controlled output devices and describes an ideal “kinesthetic display” that served as many senses as possible. Sutherland saw the potential for computers to simulate ideas and output not just visual information, but to produce meaningful sound and touch output as well, all while accepting and incorporating a user’s input in a self-modifying feedback loop. This was forward-thinking stuff; recall that when this document was written, computers weren’t even generating meaningful sounds of any real complexity, let alone visual displays capable of arbitrary content.
A Way To Experience The Unreal
What if we could similarly experience concepts that could not be realized in our physical world? It would be a way to gain intuitive and intimate familiarity with concepts not otherwise available to us.
As a first step in actualizing these ideas, Sutherland and some of his students created a large ceiling-suspended system dubbed Sword of Damocles. It was the first head-mounted display whose visuals did not come from a camera, but were generated by a computer. It displayed only line-based vector graphics, but it was able to modify what it showed in real-time based on head position and user input.
Leveraging a computer’s ability to process feedback and dynamically generate visuals was key to an interactive system capable of generating its own simulated environment. For the first time, a way to meaningfully fuse interaction with abstract simulation was within reach, and there was nowhere to go but up.
Ideas From the 60s That Happened
Many concepts that Sutherland predicted have come to pass, at least partially, and are recognizable in some modern form.
Objects Displayed by a Computer Need Not Follow Ordinary Rules of Reality
Being able to define things free from the confines of physical reality, and adjust their properties at will, encompasses functions like CAD modeling and other simulation work as well as it does entertainment like gaming. In fact, it would even be fair to say that gaming in particular thrives in this space.
Sutherland envisioned a computer-controlled display as a looking glass into a mathematical wonderland. A great example of this concept is this 3D engine for non-Euclidean geometry which presents impossible geometries in a familiar, interactive way.
Tactile and Haptic Feedback
Today’s VR controllers (and mobile devices like phones, for that matter) rely heavily on being able to deliver a range of subtle vibrations as meaningful feedback. While not measuring up to Sutherland’s ideal of accurately simulating things like complex physics, it nevertheless gives users an intuitive understanding of unseen forces and boundaries, albeit simple ones, like buttons that do not exist as physical objects.
Head Tracking
Making a display change depending on where one is looking is a major feature of VR. While Sutherland only mentioned this concept briefly, accurate and low-latency tracking has turned out to be a feature of critical importance. When Valve was first investigating VR and AR, an early indicator that they were onto something was when researchers experienced what was possible when low-persistence displays were combined with high-quality tracking; it was described as looking through a window into another world.
Hand and Body Tracking, Including Gaze Tracking
Sutherland envisioned the ability of a computer to read a user’s body as an input method, particularly high-dexterity parts of the body like the hand and eyes. Hand tracking is an increasingly common feature in consumer VR systems today. Eye tracking exists, but more on that in a moment.
Some Ideas Haven’t Happened Yet
There are a number of concepts that haven’t happened, or exist only in a very limited way, but that list is shrinking. Here are the most notable standouts.
Robust Eye Tracking Is Hard
Sutherland wrote “it remains to be seen if we can use a language of glances to control a computer,” and so far that remains the case.
It turns out that eye tracking is fairly easy to get mostly right: one simply points a camera at an IR-illuminated eyeball, looks for the black circle of the pupil, and measures its position to determine where its owner is looking. No problems there, and enterprising hackers have made plenty of clever eye tracking projects.
Eye tracking gets trickier when high levels of reliable accuracy are needed, such as using it to change how visuals are rendered based on exactly where a user is looking. There are a number of reasons for this: not only does the human eye make frequent, involuntarily movements called saccades, but roughly 1% of humans have pupils that do not present as nice round black shapes, making it difficult for software to pick out. On top of that, there is a deeper problem. Because a pupil is nothing more than an opening in the flexible tissue of the iris, it is not always a consistent shape. The pupil in fact wobbles and wiggles whenever the eye moves — which is frequently — and this makes highly accurate positioning difficult to interpret. Here is a link (cued to 37:55 in) to a video presentation explaining these issues, showing why it is desirable to avoid eye tracking in certain applications.
Simulation-Accurate Force Feedback Isn’t Ready
Force feedback devices have existed for years, and there is renewed interest in force feedback thanks to VR development. But we are far from using it in the way Sutherland envisioned: to simulate and gain intuitive familiarity with complex concepts and phenomena, learning them as well as we know our own natural world.
Simulating something like a handshake or a hug ought to be simple by that metric, but force feedback that can meaningfully simulate simple physics remains the realm of expensive niche applications that can’t exist without specialized hardware.
Holodeck-level Simulation
Probably the most frequently-quoted part of The Ultimate Display is the final few sentences, in which Sutherland describes something that sounds remarkably like the holodeck from Star Trek:
The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal. With appropriate programming such a display could literally be the Wonderland into which Alice walked.
Clearly we’re nowhere near that point, but if we ever are, it might be the last thing we ever need to invent.
Important Features vs. Cool Ones
The best ideas may have come from the 60s, but we’ve learned a lot since then about what is and isn’t actually important to creating immersive experiences. Important features are the ones a technology really needs to deliver on, because they are crucial to immersion. Immersion is a kind of critical mass, a sensory “aha!” moment in which one ceases to notice the individual worky bits involved, and begins to interact seamlessly and intuitively with the content in a state of flow. Important features help that happen.
For example, it may seem that a wide field of view in an HMD is important for immersion, but that turns out to not quite be the case. We covered a fascinating presentation about how human vision uses all sorts of cues to decide how “real” something is, and what that means for HMD development. It turns out that a very wide field of view in a display is desirable, but it is not especially important for increasing immersion. Audio has similar issues, with all kinds of things being discovered as important to delivering convincing audio simulation. For example, piping sound directly into the ear canals via ear buds turns out to be a powerful way for one’s brain to classify sounds as “not real” no matter how accurately they have been simulated. Great for listening to music, less so for a convincing simulation.
Another feature of critical importance is a display with robust tracking and low latency. I experienced this for myself the first time I tried flawless motion tracking on a modern VR headset. No matter how I moved or looked around, there was no perceptible lag or drift. I could almost feel my brain effortlessly slide into a groove, as though it had decided the space I was in and the things I was looking at existed entirely separate from the thing I was wearing on my head. That was something I definitely did not feel when I wore a Forte VFX1 VR headset in the mid-90s. At the time, it wasn’t the low resolution or the small field of view that bothered me, it was the drifty and vague head tracking that I remember the most. There was potential, but it’s no wonder VR didn’t bloom in the 90s.
What’s Next?
One thing that fits Sutherland’s general predictions about body tracking, but which he probably did not see coming, is face and expression tracking. It is experimental work from Facebook, but is gaining importance mainly for the purpose of interacting with other people digitally, rather than as a means of computer input.
Speaking of Facebook, a social network spearheading VR development (while tightening their grip on it) definitely was not predicted in the 60s, yet it seems to be next for VR nevertheless. But I never said the future of VR came from the 60s, just that the good ideas did.
“What if we could similarly experience concepts that could not be realized in our physical world? It would be a way to gain intuitive and intimate familiarity with concepts not otherwise available to us.”
I dub thee, video game. Go forth and prosper.
Video game is not the same thing as VR. Especially if we are talking about gaining skills (e.g. training) – while you can show and demonstrate something using a videogame technologies, ultimately it is still a glorified video or “powerpoint” because you are clicking on stuff using a mouse on a screen.
In VR you are actually on the scene, manipulating objects using your own hands, seeing the scene in the right proportion and scale (important e.g. for judging safe distances!) and so on. The learning impact is very different.
While there are clearly big differences, the gap between playing a first-person game on a screen with a game controller vs. using a VR headset and controllers is not necessarily so big as it may appear. When a person is focused on the screen, it really becomes their mind’s window to the game world, and when game controls have been memorized to be second nature, the thoughts to control the player are along the lines of what action is desired of the player, rather than what button to press. That is, when you want to jump, you think “jump”, and your body performs the desired action automatically, without having to process “press the upper button with your right thumb”. While it’s true that learning impact is very different, you can indeed learn from a non-VR setup as well. You can learn the space, sequence, and timing of things. You’ll have the thought to jump at the right place and time, but you’ll need to remap the actual action from button-pushing to the actual body action needed.
The human brain doesn’t work like that. Training for skills like assembling an appliance, with a screen and a mouse, doesn’t translate to the actual thing. Instead of developing muscle memory for turning your hand a certain way with certain timing, you learn the muscle memory to twitch a finger, which doesn’t help at all. Likewise, navigating 3D space on a 2D screen uses completely different senses than you would use normally.
Playing the video game, you learn the video game, not the thing you’re supposed to train for. This becomes readily apparent in things like driving games, which also attempt to incorporate physics into it, and fail spectacularly at it. People who are really good at Sega Rally tend to suck at driving an actual rally car because they’ve learned the idiosyncrasy of the game, not the real thing.
Some simulations, like car racing, translate pretty well. aside from the puking from the g forces that aren’t in the game, and the oppressive heat and smells (oil, gasoline, rubber), some racing gamers do rather well.
https://www.history.com/from-bedroom-gamer-to-professional-racer
https://www.google.com/amp/s/www.bbc.com/news/amp/newsbeat-53554245
I’m still pretty sure that I could fly a Cesna from all of my time in flight simulator growing up, but I wouldn’t dream of it without training.
Googe, ’70s Zap comixs.
There’s one use case I would like to see become reality. Going to an historical site and watching a filmed battle take place on the actual site while wearing the goggles.
That would be incredibly cool. Take all the data and film available, make a full simulation of the battle, and be able to hop around from ‘hot spot’ to ‘hot spot’ that had video or other evidence from that point. You could take a passive guided tour, or you could pause the video from one part of the battle, zoom out overhead, find a different spot and watch the video from there. Or zoom out to a high overhead view, and watch the battle in fast forward. Of course, you’d have to take some artist’s liberties, but I think you could make it work.
And so costly that no museum would be able to afford it. People don’t realize that you can’t just take some video and turn into a “simulation” (i.e. something interactive beyond jumping between premade points of view).
Never mind the practicalities of loaning fragile goggles to throngs of people – the goggles need to be cleaned, charged, repaired constantly and replaced. Even without the entire COVID mess that would be a business nightmare.
lol, I’m missing the part where I said it would be easy… I work in the industry, I understand how hard it would be. People would pay good money for a product like this. It could also be wrapped and packaged as some sort of game which might make it an easier sell.
I also missed the part where anyone said anything about a museum?
Historical site is not a museum? Or do you expect people to bring their own device, download and app and go walk somewhere in the field? I wonder which device would that be?
You are also seriously overestimating the amount someone would be willing to pay for this (and underestimating how much would a production like this actually cost, given that it would be a one-off completely bespoke thing).
People willing to pay “good money” often stop being willing to pay the moment you announce the price. Even stupid “e-learning”/360 video (which is not virtual reality by any stretch) style content using off-the-shelf tooling would cost somewhere between $20k-$100k to produce – and that would be still cheap. Truly interactive content would be even more expensive.
Are you willing to sponsor that? Because it won’t happen otherwise.
I’ve thought a lot about this. I think you would have to have reenactors recreate the battle on the actual field. Of course Gettysburg is the battle that would attract the most people.
It’s actually a trick. There are no good VR ideas.
I dunno, I’ve dabbled with VR over the years, but with the Quest 2, it finally feels like it is ready for prime-time. I’m having a lot of fun with mine.
You know, if there was an actual need for seeing 3D graphics from a computer, we would be already using CAD programs with those stupid 3D glasses, as the cheapest way of doing that. But nobody ever does it. Not because it’s had to do, or not good enough, but simply because it’s unnecessary — a flat projection is more than enough for work. Sure, new toys may seem cool and may even give you a breathtaking experience or two sometimes, but they simply have no use beyond that.
Except for the fact that just about every 3D modeling system has extensive features for displaying 3D images of designs for viewing by clients. Maybe designers can work on 3D designs in 2D but it’s pointless without happy customers to pay for it.
“Nobody ever does it” that is a baldfaced lie.
Interesting, can you give an example of a CAD program that uses 3D glasses? Obviously I haven’t seen every single program in existence, but I’ve used quite a few, and seen a few more being used by others, and never saw such an option.
You need 2d glasses because I never said that cad programs come with 3d glasses.
If the application benefits from a sense of immersion, then VR is really good. CAD is an example of something that really doesn’t benefit from greater immersion. However, VR could be useful in displaying designs to customers could be useful. I imagine that there are many training applications for VR.
And yes. VR makes for a rather good toy.
CAD/CAM mostly benefits from 3D in the visualization department, because of the way the objects are defined as projections on 2D planes.
Most of your work in CAD is making a flat constrained line drawing of your part, which you then extrude out. There are various “3D drawing” tools, but those result in poorly defined geometries that are difficult to manufacture.
Is CAD 2.5D b/c of the 2D display, or does the causality go the other way?
Pretty sure ANSYS2020 has VR support
The article is completely ignoring the work of Myron Krueger which has started in late 60s/early 70s:
https://en.wikipedia.org/wiki/Myron_W._Krueger
He did not work with gloves and goggles style of VR because at the beginning it didn’t exist and when it finally did (early late 80s, early 90s) he didn’t like how encumbering the user the equipment was (and still is) – in fact, he called that the “monkey suits” (there was that concept of “datagloves” and “datasuits” back in the 90s).
His work has always been using video – large projection systems, light, using video cameras for interaction. Decades before people like Johnny Lee were showing two handed interaction/multitouch, Kinect, or even using computers for graphics – his work was using classic video production techniques like green screens and video mixers.
In fact, he didn’t even use the term virtual reality – he called his work “artificial reality” to explicitly distance himself from the “goggles & gloves” crowd that was emerging at the time.
He has published a book “Artificial Reality” which is worth reading – a lot of the stuff in it was very visionary for the time and we know it today in different forms. Also googling stuff like “Small World”, “Videoplace” and “Psychic space” should give you some information on his work.
Speaking of old VR ideas imagine if these two things came to be:
1) A port to modern hardware/OSs of GopherVR https://en.wikipedia.org/wiki/GopherVR
and
2) A Gopher interface to HaD
There’s already HaD retro edition…
Or a re-implementation of GopherVR. I meant to write that too before submitting. Those graphics were pretty boring by today’s standards.
Remember 3D filesystem browsers? Oh man that was the best idea back in the late 90s…
(No idea why.)
LOL. Definitely no practical purpose but perfect for someone demoing their brand new cyberdeck build.
preparing the human race for vr life
Humans have been making virtual environments since the first cave drawings, humans live in VR all the time. Have you ever heard of “fiction”??? Hint: it’s not real.
Too many drugs!
” In VR you are actually on the scene, manipulating objects using your own hands”
Someone really needs to look up the words “actual” and “virtual” in the dictionary.
I worked for a company that did holodeck type VR, so no goggles etc., you were in a big dome with and image projected onto it. It was used for weapons training, shoulder launched missiles etc. You can still do stereoscopic images with shutter glasses.
Tilt 5 is making a headset which has that kind of tech but meant for a tabletop surfaces. There’s talk among the Tilt 5 developers discord about creating a holodeck like experience but nobody really knows for sure if it’s going to work for that purpose.
I really recommend movies or books like “World on a Wire” (Welt am Draht), 13th Floor, Simulacron-3 (aka Counterfeit World)..
https://en.wikipedia.org/wiki/World_on_a_Wire
The Sensorama was invented and patented by Morton Heilig: https://patents.google.com/patent/US3050870A/en
He also invented and patented other VR related devices. An HMD from back in that era was his Telesphere Mask: https://www.techradar.com/news/wearables/forgotten-genius-the-man-who-made-a-working-vr-machine-in-1957-1318253/2
Perhaps his most interesting patented invention was his Experience Theater: https://patents.google.com/patent/US3469837A/en
VR has the potential to revolutionize the way we see the world! – imagine traveling back in time to a distant past in history – walking with the dinosaurs a million years ago, or walk around an ancient city? Maybe walk on the surface of the Moon or Mars as though you were really there? swim with the dolphins or fly with an eagle! – Just like computer simulations of 2D content for movies, as we approach the point that it becomes more and more difficult to tell the difference between a “real” scene , or a CGI generated scene, VR will become so good, that it will eventually become difficult to tell the difference between the real world or a computer generated world! – I am really excited by the potential of VR and hope to see more progress with the technology!
I’d say thing like AR and MR would be a greater force for change. VR is kind of tied down due to it’s nature.
^ This.
I’ve long wished I could get a decent helmet/heads up AR display with a thermal image I can use while driving to help dodge all the suicidal deer and elk around here.
The reason earbud sound doesn’t seem real is that the external ear produces important phase and volume information that the brain interprets for sound source direction, and to sort out competing sounds by their location.
VR in the 60s/70s had spot-on concepts but no way to practically attempt to implement them (TRL1/2). VR in the 90s had implementations but they were impractical and scattered between multiple independent demo systems (TRL3/4), but were literally not capable of producing a system similar to todays VR systems regardless of the available funding. In the early 2010s there started a rapid convergence of demonstrated features into single devices/systems at commodity cost and produced en mass (e.g. DK1 & DK2 ~ TRL6/7). Today (2020s) we are up to TRL9 with commodity devices that implement all or nearly all previously demonstrated features.
Analogously, AR today is in a similar state to VR in the 90s: we have a pretty good idea of what AR will look like and how it should function. We have some single-feature demos. We are decades away from practical devices that can do all those things, let alone do them at any sort of non-one-off cost. There are some basic demonstrations of technologies that /might/ be able to solve some of AR’s specific problems (occlusion of real-world objects by virtual ones and vice versa, real and virtual objects existing in the same range of coal planes, render loops that have to beat real-world photons rather than just to 20ms motion-photons budget of VR, etc) but we could literally not build a ‘set of AR glasses’ that achieves what we know AR needs to be able to do regardless of funding.
The current prophesied ‘AR boom’ will be as transient as the VR boom of the 90s was.