Google’s Augmented Reality Microscope Might Help Diagnose Cancer

Dr. Niels Olson uses the Augmented Reality Microscope. (Credit: US Department of Defense)

Despite recent advances in diagnosing cancer, many cases are still diagnosed using biopsies and analyzing thin slices of tissue underneath a microscope. Properly analyzing these tissue sample slides requires highly experienced and skilled pathologists, and remains subject to some level of bias. In 2018 Google announced a convolutional neural network (CNN) based system which they call the Augmented Reality Microscope (ARM), which would use deep learning and augmented reality (AR) to assist a pathologist with the diagnosis of a tissue sample. A 2022 study in the Journal of Pathology Informatics by David Jin and colleagues (CNBC article) details how well this system performs in ongoing tests.

For this particular study, the LYmph Node Assistant (LYNA) model was investigated, which as the name suggests targets detecting cancer metastases within lymph node biopsies. The basic ARM setup is described on the Google Health GitHub page, which contains all of the required software, except for the models which are available on request. The ARM system is fitted around an existing medical-grade microscope, with a camera feeding the CNN model with the input data, and any relevant outputs from the model are overlaid on the image that the pathologist is observing (the AR part).

Although the study authors noted that they saw potential in the technology, as with most CNN-based systems a lot depends on how well the training data set was annotated. When a grouping of tissue including cancerous growth was marked too broadly, this could cause the model to draw an improper conclusion. This makes a lot of sense when one considers that this system essentially plays ‘cat or bread’, except with cancer.

These gotchas with recognizing legitimate cancer cases are why the study authors see it mostly as a useful tool for a pathologist. One of the authors, Dr. Niels Olsen, notes that back when he was stationed at the naval base in Guam, he would have liked to have a system like ARM to provide him as one of the two pathologists on the island with an easy source of a second opinion.

(Heading image: Dr. Niels Olson uses the Augmented Reality Microscope. (Credit: US Department of Defense) )

10 thoughts on “Google’s Augmented Reality Microscope Might Help Diagnose Cancer

  1. Well, I hope Google’s CNN is more accurate in its reporting than the television channel with the same initials.
    But then again, it is Google.

  2. A cancer diagnosis is always a pathological/histological diagnosis. Many/mostnow also confirm with molecular genetic, flow cyto and advanced staining and other methods. Not just for diagnosis but prognosis and treatment options.
    That said yeah first step is always the old fashioned microscope.

  3. Friend works in a cancer lap, he said that cancer cells are visible different from the sorounding onces. Like the cancer cell looks more fatty and have cellutite compared to the other round shapes. He looks in the miscroscope and counts how many cancer cells are there, so he can give feedback if the therapy works (cancer cells goes down) or nothing happend (the cancer cells amount stay same).
    So for what did we need A.I.?

    1. Erm, so your well qualified friend can do something better with her PhD qualified ass than simply counting circles? Maybe, free her up to do something more valuable?
      I mean, I get that luddites don’t want automation but this seems like a odd choice of application to object to.

    2. Cancer cells do often look very different, but in a sample there is already a bunch of “odd” things, like blood vessels. This tool can, for example, tell the pathologist “hey, look at this corner, there is this one cell that looks funny”. Something that a human might miss, and it is of real concern in the clinic. Basically, the technology can, if it works well enough, make human experts faster and more accurate.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.