Putting on a headset and using virtual monitors in VR instead of physical ones is a use case that pops up, but is it really something feasible? [Karl Guttag], who has long experience and a deep understanding of the technical challenges that face such devices, doesn’t seem to think so.
In his writeup [Karl] often focuses on the recently-unveiled high resolution Apple Vision Pro, but the issues he discusses transcend any particular product. His article is worth the read for anyone with an interest in these issues, but we’ll summarize some main points here.
A primary job of a monitor is to display text. Text is so important, in fact, that it gets handled as a special case. Font hinting is the process (and art) by which the fine details and lines inherent to text are translated to a pixel grid for best appearances in all circumstances. This grid fitting — specialized mapping of characters to fit a rasterized grid of pixels — has been a major part of computers for over forty years, and almost all font rendering relies on it. It’s an especially important part of making text look good on lower resolution displays.
But normal grid fitting cannot be applied to a virtual monitor because one cannot rely on content being rendered in any particular orientation. Between a monitor and a virtual display of equal resolution, virtual text will always look worse because the usual grid fitting tricks don’t work. Giving a virtual monitor higher resolution is one way to fix this, but at least so far, virtual monitors are going to have fewer pixels to work with, not more. Physical monitors do not have to share their pixels with anything else, but the little screens inside VR headsets do.
Making virtual monitors bigger is another way to give them more pixels, but as virtual monitors get bigger they get less comfortable to use. As soon as one begins to have to move one’s head to read the whole screen, things get very tiresome very quickly. For serious desktop computer work, movie theatre-sized virtual monitors are not ideal.
[Karl] seems to have good reasons to believe VR headsets will continue to make poor monitor replacements, but there are other efforts from people and organizations trying to make it work. The SimulaVR folks, for example, aim to make an open-source Linux headset that functions primarily as a monitor replacement, and their proposed solution to boost resolution is a lens that squashes pixels towards the center of the field of view. Another direction is from a company unsatisfied with existing headsets’ ability to render virtual monitors, so they announced their own. Neither of those are finished products one can actually buy at the moment, but clearly [Karl] isn’t the only one thinking about these problems.
Would you use a VR headset to replace your monitor? Either way, such a headset would probably make a good basis for a cyberdeck build.
This seems to ignore that Apple doesn’t do hinting and removed subpixel font smoothing a few years ago?
No need for subpixel smoothing when your pixels are small enough, I guess?
“Retina” VR headsets will presumably solve most of these issues. But they’ll still be heavy and bulky for a long time.
This is entirely dependent upon the headset. If the image has been stretched and distorted to fill a larger viewport, sure. Its ALWAYS GOING TO SUCK.
When the resolution of displays, and the computing power to feed them has sufficiently grown, This issue will be of much less concern.
Until then, Stick with waveguide equipped AR headsets like Epson Moverio or lenovo’s thinkreality A6 (not the birdbathed a3). Hopefully, Lumus will get their Zlens into something consumer grade in 2024.
The guy knows his stuff, but on the other hand, Apple is fairly clever, and the device isn’t cheap.
Pixel per degree of field of view is everything that matters.
I don’t think that text rendering is going to be the reason it sucks. Maybe one of the reasons, but the big one is that you can plug and play a monitor, and you can’t for VR. And nobody is ever going to make a decent one without some awful walled garden of proprietary software that has to come with it and manage everything that happens through it. We are in the age of megabloat subscription software now, and they will all have it because the development cost doesn’t make sense without it
“Nobody is ever going to…” – well that’s a dumb thing to say. Unless you have a crystal ball with a subscription to all the tech news that will ever happen you cannot possibly justify that assertion, it’s on a par with “nobody will ever need more than 640k”.
You can just use a VR headset as a monitor if you want to on PC, though outside of a few weird use cases its not the best idea. That really isn’t a big problem, at least in its own right – it might be when somebody creates a really good VR optic/screen combo for desktop use and deliberately tries to make it impossible to use outside of their walled garden.
Really I have to agree with the article until text can be rendered at a sensible scale and be easily readable VR as a desktop replacement will be a very very niche thing. Great for some folks I’m sure, perhaps the folks that spend most of their time computing creating and consuming 3d models and video anyway using their phone for general communication tasks like email. But its going to suck for most folk as you don’t want to have to move your head to the sweet spot where the word you are trying to read gets big enough to render crisply.
>>And nobody is ever going to make a decent one without some awful walled garden of proprietary software that has to come with it and manage everything that happens through it.
You just described Apple :-(
Seriously, though, it might not be too difficult for a company to build a plug-and-play type system. For an external monitor, you could have a physical device with an hdmi or displayport input feeding a data converter (can’t really say a rasterizer, the data is already digital) that transmits to the headset (its the thing that has to deal with mapping the display to the user’s view so it does the heavy lifting). There would be lag, but maybe only a frame or two (this kind of setup wouldn’t be for high-performance gaming anyway… those types of games would work directly with a headset). Might even be able to support DRM/HDCP, at least for IOS / Windows stacks. Yes, that brings in the walled garden, but while the software would be proprietary, it might not be disrupted for those already neck deep in Apple or Android Land.
If the major players thought there would be a market (I don’t think they would care about desktops… the money is in “content consumers”), I could see virtual monitor extensions hooking into RDP for windows and whatever Apple uses for remote desktops. I regularly use VNC to remote a virtual desktop to an Android phone, its not that much different. I don’t know how GPUs would apply to a virtual monitor, I could see the driver “redirecting” video content that would be going out a physical port. Again, the GPU manufacturers would have to believe there was enough of a market to make this worthwhile.
Hm. VR has at least the potential for proper simulation of CRT monitors, eventually..
The 3d models available online of the C64, NES and the slightly overrated Commodore 170x monitor aren’t bad it seems.
There’s EmuVR, for example; very interesting.
It can even run Windows 95 on a virtual monitor.
https://www.core77.com/posts/82315
https://www.youtube.com/watch?v=YehG_q8X0uY
he makes the assumption that the small displays will have fewer pixels to work with, and that they won’t be a grid. I don’t see why those assumptions are going to be the case. There’s no reason why the display won’t be a grid, and the cost of a display is determined by not just the resolution, but also the size.
Yes, at some point making smaller transistors/LEDs raises the cost, but above that point having a large area means that you are more likely to have a flaw in the display and have to discard it (your yield goes down),
I share the concerns about a walled garden around the first versions of this, but once they get out into the wild you will have some hackers reverse engineering them, and shortly after you will have cheap clones being produced, so I expect that wall to fall fairly quickly.
VR displays *already* aren’t in a proper grid. The panels they use will be, but the optics severely warp the display and require pre-compensation (this is the design that Oculus pioneered which kickstarted the modern VR revolution.) Even on other headsets that are more flat (nreal/rokid/etc which use micro-oled displays and birdbath optics) you still have the problem that the text isn’t being rendered 1-1 on that display, it’s being mapped to a virtual surface which has arbitrary orientation in the view.
The real reason that it’s not necessarily a big problem is that a virtual display is not restricted in “physical” size, so you don’t *have* to squeeze the text as small as possible to fit enough into view. Just make it big enough to be comfortably legible, and make the virtual display larger!
of course, anyone who is significant long sighted (I need glasses to be able to focus on things that are even 1m away) VR are going to be pretty much a non starter anyway…
As a person who can’t focus farther than ~0.3 meters, like it’s a competition :D, I disagree. Prescription lenses are available for many headsets and if one is comfortable with contact lenses, they are an option too.
In a hackaday fashion I adapted a pair of cheap (zennioptical) aviator style large lens glasses to snap-fit to my Pimax headset. Soldered lengths of ~1mm steel wire to top and bottom of each lens frame and bent them to _(‾)_ shape so they snap to the faceplate of the headset. The lenses are held very close to the optics of the headset, giving largest gap to brow to minimize oil and sweat transfer from eyebrows. (imo that generation headset is still very far from being good for ” full-day” use , resolution or comfort wise, they are reserved for simulator gaming and some visualisation work for me)
Great, now we’ll be able to cram twice as people in those call centers
VR is probably not a great option if you’re working on a spreadsheet. But VR may be just fine for watching a PowerPoint presentation on a Zoom call. AR, (Augmented Reality) which lets you see both the computer image and the world around you, is very useful right now. My buddy has an XREAL Air which he raves about for attending meetings. The only downside is that the Air requires a wire tether. He has it connected to a Pi with a battery pack, both attached to his belt. He can walk around his house while attending a meeting without missing a thing. So yeah, text is important and a headset isn’t a replacement for a monitor for most people right now. But they can be a very useful addition to your current setup.
Seems to me that drawing a picture of a monitor in your VR goggles is kind of like drawing a picture of a pen and paper on a monitor. If you want to display text, find a “native” way to display text…
Don’t forget using your keyboard. Sure, a good typist is going to know where all the keys are but if you stop using your keyboard for a while you have to grope for it and then reorient yourself and then backspace over everything that you accidentally typed before you oriented.
Will augmented reality solve that issue? Yeah I don’t know maybe, but I don’t know who’s going to solve the issue of having sweaty raccoon face and next teague of wearing a big doofy thing on your head for 6 hours a day.
And the wave guide AR stuff sucks too. 10° field of view or whatever it is? Sad, weird rainbows as seen through a toilet paper tube.
Neck fatigue. And I’ll go ahead and use this as an example to point out that voice typing isn’t going to solve this input problem either.
Neck fatigue is only an issue for the moment. The displays will get lighter. I don’t hear any complains about the weight of audio headsets. Eventually MR headsets will get to a reasonable weight range for all users. (Or we just keep moving things off the head into a brick, a la Apple Vision’s battery)
The battery’s moved on the apple version because they insisted on it being metal and glass.
I feel like if they wanted to keep it light but provide that “i overpaid” expensive-feel, they probably should’ve made it out of a lightweight but ritzy wood. Glass is already fragile so strength wouldn’t be an issue and neither would moisture because I’m certain Apple isn’t going to suggest you use them in a rain storm.
There are already very small lightweight headsets coming out that don’t have batteries or all the processing on board because they’re meant to just be displays hooked to a computer. (The bigscreen beyond is one, but i’m there are others) Also in my experience the neck fatigue doesn’t come from the weight, but the balance. My quest gets uncomfortable for me in 45 minutes or so, Unless I strap a big battery to the back that is close to doubling the weight. Then it’s comfortable to use for hours.
My thought as well. Use of keyboard, peripherals, situational awareness around you — all are gone with a head set. No I don’t ever see a headset in my future. That said, hologram like ‘screens’ (one or multiple) that just pop up in front of your keyboard that you could position as needed… Now that seems a better way to go, but a tougher tech challenge. Seems like then a whole area above your work desk could be used as your visual interface.
I don’t like wearing devices when I am trying to be productive at my workstation. I tried headsets and earbuds for years on video calls and always found them to be uncomfortable for long periods of time. Eventually I switched them all for an office speakerphone so I can stand up and move around freely while I’m attending meetings.
Virtual displays within the VR display only make sense if you want to use VR as a training tool. Think like showing a retail employee how to operate and demonstrate a real TV, or replicating the LCD readout from a helicopter cockpit. There’s no point to using it to replicate consumer monitors for productivity because consumer monitors are superior, cheap, and abundant.
I think Apple’s best idea is to take the battery out of the headset. Would’ve been better to also take out the computer, but hey. Makes much more sense to me to keep the thing you strap to your face as light-weight as possible (and not have it contain a component that’s one bad day away from exploding), and put everything else in your pocket or on a belt clip or something.
Aside from that, there’s already plenty of evidence showing that sitting inside looking at a monitor screws up your eyes, I’m guessing that using VR for office work is going to make that way worse, much as I think it would be cool to work like one of the air traffic controllers in The Matrix Reloaded.
I used my Quest 3 as a monitor this week working as a software developer, and plan to continue.
I found it better than using a laptop monitor, since my screen is huge in VR and I can re-orient it as I move to different work spaces in my office.
I also turn the laptop’s brightness to 0 so the laptop’s battery lasts longer.
Sure, sometimes I zoom in to make the text larger, but the text is super readable and the headset is light enough that I can wear it for 8-10 hours a day without a problem.