Meta Cancels Augmented Reality Headset After Apple Vision Pro Falls Flat

The history of consumer technology is littered with things that came and went. For whatever reason, consumers never really adopted the tech, and it eventually dies. Some of those concepts seem to persistently hang on, however, such as augmented reality (AR). Most recently, Apple launched its Vision Pro ‘mixed reality’ headset at an absolutely astounding price to a largely negative response and disappointing sale numbers. This impending market flop seems to now have made Meta (née Facebook) reconsider bringing a similar AR device to market.

To most, this news will come as little of a surprise, considering that Microsoft’s AR product (HoloLens) explicitly seeks out (government) niches with substantial budgets, and Google’s smart glasses have crashed and burned despite multiple market attempts. In a consumer market where virtual reality products are already desperately trying not to become another 3D display debacle, it would seem clear that amidst a lot of this sci-fi adjacent ‘cool technology,’ there are a lot of executives and marketing critters who seem to forego the basic question: ‘why would anyone use this?’

Continue reading “Meta Cancels Augmented Reality Headset After Apple Vision Pro Falls Flat”

A Closer Peek At The Frame AR Glasses

The Frame AR glasses by Brilliant Labs, which contain a small display, are an entirely different approach to hacker-accessible and affordable AR glasses. [Karl Guttag] has shared his thoughts and analysis of how the Frame glasses work and are constructed, as usual leveraging his long years of industry experience as he analyzes consumer display devices.

It’s often said that in engineering, everything is a tradeoff. This is especially apparent in products like near-eye displays, and [Karl] discusses the Frame glasses’ tradeoffs while comparing and contrasting them with the choices other designs have made. He delves into the optical architecture, explaining its impact on the user experience and the different challenges of different optical designs.

The Frame glasses are Brilliant Labs’ second product with their first being the Monocle, an unusual and inventive sort of self-contained clip-on unit. Monocle’s hacker-accessible design and documentation really impressed us, and there’s a pretty clear lineage from Monocle to Frame as products. Frame are essentially a pair of glasses that incorporate a Monocle into one of the lenses, aiming to be able to act as a set of AI-empowered prescription glasses that include a small display.

We recommend reading the entire article for a full roundup, but the short version is that it looks like many of Frame’s design choices prioritize a functional device with low cost, low weight, using non-specialized and economical hardware and parts. This brings some disadvantages, such as a visible “eye glow” from the front due to display architecture, a visible seam between optical elements, and limited display brightness due to the optical setup. That being said, they aim to be hacker-accessible and open source, and are reasonably priced at 349 USD. If Monocle intrigued you, Frame seems to have many of the same bones.

Two pictures of the same black dog, wearing two separate pairs of the AR glasses reviewed in these two articles

A Master-Class On Reverse-Engineering Six AR Glasses

Augmented reality (AR) tech is getting more and more powerful, the glasses themselves are getting sleeker and prettier, and at some point, hackers have to conquer this frontier and extract as much as possible. [Void Computing] is writing an open source SDK for making use of AR glasses, and, along the way, they’ve brought us two wonderful blog posts filled with technical information laid out in a fun to read way. The first article is titled “AR glasses USB protocols: the Good, the Bad and the Ugly”, and the second one follows as “the Worse, the Better and the Prettier”.

Have you ever wanted to learn how AR glasses and similar devices work, what’s their internal structure, which ones are designed well and which ones maybe not so much? These two posts have concise explanations, more than plenty of diagrams, six case studies of different pairs of AR glasses on the market, each pair demonstrated by our hacker’s canine assistant.

[Void Computing] goes in-depth on this tech — you will witness MCU firmware reverse-engineering, HID packet captures, a quick refresher on the USB-C DisplayPort altmode, hexdumps aplenty, and a reminder on often forgotten tools of the trade like Cunningham’s law.

If reverse-engineering lights your fire, these high-level retrospectives will teach you viable ways to reverse-engineer devices in your own life, and they certainly set a high bar for posts as far as write-ups go. Having read through these posts, one can’t help but think that some sort of AR glasses protocol standard is called for here, but fortunately, it appears like [Void Computing]’s SDK is the next best thing, and their mission to seize the good aspects of a tentative cyberpunk future is looking to be a success. We’ve started talking about AR glasses over a decade ago, and it’s reassuring to see hackers catching up on this technology’s advancements.

We thank [adistuder] for sharing this with us on the Hackaday Discord server!

Hackaday Links Column Banner

Hackaday Links: February 11, 2024

Apple’s Vision Pro augmented reality goggles made a big splash in the news this week, and try as we might to resist the urge to dunk on them, early adopters spotted in the wild are making it way too easy. Granted, we’re not sure how many of these people are actually early adopters as opposed to paid influencers, but there was still quite a bit of silliness to be had, most of it on X/Twitter. We’d love to say that peak idiocy was achieved by those who showed themselves behind the wheels of their Teslas while wearing their goggles, with one aiming for an early adopter perfecta, but alas, most of these stories appear to be at least partially contrived. Some people were spotted doing their best to get themselves killed, others were content to just look foolish, especially since we’ve heard that the virtual keyboard is currently too slow for anything but hunt-and-peck typing, which Casey Niestat seemed to confirm with his field testing. After seeing all this, we’re still unsure why someone would strap $4,000 worth of peripheral-vision-restricting and easily fenced hardware to their heads, but hey — different strokes. And for those of you wondering why these things are so expensive, we’ve got you covered.

Continue reading “Hackaday Links: February 11, 2024”

Dr. Niels Olson uses the Augmented Reality Microscope. (Credit: US Department of Defense)

Google’s Augmented Reality Microscope Might Help Diagnose Cancer

Despite recent advances in diagnosing cancer, many cases are still diagnosed using biopsies and analyzing thin slices of tissue underneath a microscope. Properly analyzing these tissue sample slides requires highly experienced and skilled pathologists, and remains subject to some level of bias. In 2018 Google announced a convolutional neural network (CNN) based system which they call the Augmented Reality Microscope (ARM), which would use deep learning and augmented reality (AR) to assist a pathologist with the diagnosis of a tissue sample. A 2022 study in the Journal of Pathology Informatics by David Jin and colleagues (CNBC article) details how well this system performs in ongoing tests.

For this particular study, the LYmph Node Assistant (LYNA) model was investigated, which as the name suggests targets detecting cancer metastases within lymph node biopsies. The basic ARM setup is described on the Google Health GitHub page, which contains all of the required software, except for the models which are available on request. The ARM system is fitted around an existing medical-grade microscope, with a camera feeding the CNN model with the input data, and any relevant outputs from the model are overlaid on the image that the pathologist is observing (the AR part).

Although the study authors noted that they saw potential in the technology, as with most CNN-based systems a lot depends on how well the training data set was annotated. When a grouping of tissue including cancerous growth was marked too broadly, this could cause the model to draw an improper conclusion. This makes a lot of sense when one considers that this system essentially plays ‘cat or bread’, except with cancer.

These gotchas with recognizing legitimate cancer cases are why the study authors see it mostly as a useful tool for a pathologist. One of the authors, Dr. Niels Olsen, notes that back when he was stationed at the naval base in Guam, he would have liked to have a system like ARM to provide him as one of the two pathologists on the island with an easy source of a second opinion.

(Heading image: Dr. Niels Olson uses the Augmented Reality Microscope. (Credit: US Department of Defense) )

Beautifully Rebuilding A VR Headset To Add AR Features

[PyottDesign] recently wrapped up a personal project to create himself a custom AR/VR headset that could function as an AR (augmented reality) platform, and make it easier to develop new applications in a headset that could do everything he needed. He succeeded wonderfully, and published a video showcase of the finished project.

Getting a headset with the features he wanted wasn’t possible by buying off the shelf, so he accomplished his goals with a skillful custom repackaging of a Quest 2 VR headset, integrating a Stereolabs Zed Mini stereo camera (aimed at mixed reality applications) and an Ultraleap IR 170 hand tracking module. These hardware modules have tons of software support and are not very big, but when sticking something onto a human face, every millimeter and gram counts.

Continue reading “Beautifully Rebuilding A VR Headset To Add AR Features”

Chatting About The State Of Hacker-Friendly AR Gear

There are many in the hacker community who would love to experiment with augmented reality (AR), but the hardware landscape isn’t exactly overflowing with options that align with our goals and priorities. Commercial offerings, from Google’s Glass to the Microsoft HoloLens and Magic Leap 2 are largely targeting medical and aerospace customers, and have price tags to match. On the hobbyist side of the budgetary spectrum we’re left with various headsets that let you slot in a standard smartphone, but like their virtual reality (VR) counterparts, they can hardly compare with purpose-built gear.

But there’s hope — Brilliant Labs are working on AR devices that tick all of our boxes: affordable, easy to interface with, and best of all, developed to be as open as possible from the start. Admittedly their first product, Monocle, it somewhat simplistic compared to what the Big Players are offering. But for our money, we’d much rather have something that’s built to be hacked and experimented with. What good is all the latest features and capabilities when you can’t even get your hands on the official SDK?

This week we invited Brilliant Lab’s Head of Engineering Raj Nakaraja to the Hack Chat to talk about AR, Monocle, and the future of open source in this space that’s dominated by proprietary hardware and software.

Continue reading “Chatting About The State Of Hacker-Friendly AR Gear”