The cameras at the front of Meta’s Quest VR headsets are off-limits to developers, but developer [Michael Gschwandtner] created a workaround (Linkedin post) and shared implementation details with a VR news site.
The demo shows object detection via MobileNet V2, which we’ve seen used for machine vision on embedded systems like the Raspberry Pi. In this case it is running locally on the VR headset, automatically identifying objects even though the app cannot directly access the front-facing cameras to see what’s in front of it.
The workaround is conceptually simple, and leverages the headset’s ability to cast its video feed over Wi-Fi to other devices. This feature is normally used for people to share and spectate VR gameplay.
First, [Gschwandtner]’s app sets up passthrough video, which means that the camera feed from the front of the headset is used as background in VR, creating a mixed-reality environment. Then the app essentially spawns itself a Chromium browser, and casts its video feed to itself. It is this video that is used to — in a roundabout way — access what the cameras see.
The resulting view isn’t really direct from the cameras, it’s akin to snapshotting a through-the-headset view which means it contains virtual elements like the UI. Still, with passthrough turned on it is a pretty clever workaround that is contained entirely on-device.
Meta is hesitant to give developers direct access to camera views on their VR headset, and while John Carmack (former Meta consulting CTO) thinks it’s worth opening up and can be done safely, it’s not there yet.
It’s ironic how they lock camera for devs, but they have all the access like “trust us bro”. I understand why they lock it, but it feels weird. I will be hacking my headset if I ever go back to Meta. At least with index I’m able to have control over the camera feed.
The whole world would have been a little bit better if oculus was bought by a game developer, instead of a company that uses privacy abuse as it’s main core business.
A little search shows that the kickstarter for the Oculus was in 2012.
https://www.kickstarter.com/projects/1523379957/oculus-rift-step-into-the-game
This also makes me wonder how Jeri Ellsworth is doing with her https://www.tiltfive.com/ project. Short survey shows it’s still alive but it seems to be slow going. I do wish her well, and at least there are no privacy issues with it’s “front viewing camera’s”. Those are still the same old biological gadgets.
Yeah, Meta doesn’t allow users access to the cameras, but what do they do with the images?
Sell them to the highest bidder as usual?
How times have changed. Back in the Cold War era espionage devices were free for the target, including installation and maintenance…
They also were in a joint effort with valve R&D for VR. They then ran off with said research leaving valve to build from scratch, and coming up with the genius, and far superior when it comes to precision, light house tracking!.
What, pray tell, are they there for then? The quality is faaaar in excess of what’s needed for tracking.
Oh, right. To help Facebook serve you more accurate ads by collecting contextual clues from your surroundings… Silly me!
Expect to get ads for oversized nighties if the lady of the house goes out of shape. Or a visit by guys you don’t want to meet if you have suspicious things lying around.
person who does SLAM here. Those RGB cameras are not used for tracking (well not quite, they kinda can be,) They are used to re-project the colour world outside into your eyes. They need to be high enough quality so that they can extra depth (the depth cameras are rarely if ever used) so that warping around hands and objects near your face is minimised. They also need to be super high qulity so you can see what you’re interacting with.
Try reading a text message on your phone in MR. Congrats, its super hard. From what I;ve been told MR on the quest 3 is almost like being legally blind in the UK. I’m not sure how true that is.
Carmack speculates about apps recording your spouse walking around the house naked, and his worry is that people will use that to criticise Meta, for which his solution is to “be brave”.
IMO this is your daily reminder that being a genius in one field doesn’t stop you being a dangerous cretin in others. I’m glad he’s not making the decisions about privacy hazards, and that Meta have maybe learned some belated lessons.
When / if AR headsets become a real prospect, apps will need camera access, but unless that comes with safeguards that make iOS look permissive, from day one, it will be an industry-killing liability nightmare.
Given that current headsets can’t support useful AR anyway, I think Meta are wise to block camera access, and should probably treat this hack as a vuln. Nothing against the hack itself, but the use cases are mostly horrible.
Local camera access disabled or not, ALL AR goggles that are not fully self-contained and need to access an external server are a security and privacy nightmare – and possibly illegal in countries that have real privacy laws.
Why so? Do we really need to be prevented hardware access to cameras?
The use cases are mostly horrible? They are cameras. People will do people things with them and if they break any laws we have some in place to deal with it.
Geez, what’s with everyone playing bumper life all of a sudden? Buying a camera that sees everything so that it can prevent you from seeing everything..
VR was meant to be the next big thing. That’s why Meta invested in it. Fortunately it all fizzled out like 3D TVs.