Hey Google, Is My Heart Still Beating?

University of Washington researchers studying the potential medical use of smart speakers such as Amazon’s Echo and Google’s Nest have recently released a paper detailing their experiments with non-contact acoustic heartbeat detection. Thanks to their sensitive microphone arrays, normally used to help localize voice commands from the user, the team proposes these affordable and increasingly popular smart home gadgets could lead a double life as unobtrusive life sign monitors. The paper goes so far as to say that even with multiple people in the room, their technique can be used to monitor the heart and respiratory rate of a specific target individual.

Those are some bold claims, but they aren’t without precedent. Previous studies performed at UW in 2019 demonstrated how smart speaker technology could be used to detect cardiac arrest and monitor infant breathing. This latest paper could be seen as the culmination of those earlier experiments: a single piece of software that could not just monitor the vitals of nearby patients, but actually detect a medical emergency. The lifesaving potential of such a program, especially for the very young and elderly, would be incredible.

So when will you be able to install a heart monitor skill on the cheap Echo Dot you picked up on Prime Day? Well, as is often the case with this kind of research, putting the technique to work in the real-world isn’t nearly as easy as in the laboratory. While the concept is promising and is more than worthy of further research, it may be some time before our lowly smart speakers are capable of Star Trek style life-sign detection.

Can You Hear Me Now?

The researchers explain in the paper that their work is built on earlier experiments that used powerful ultrasound transducers to detect minute displacements of a patient’s chest, abdomen, and neck. Relatively large motions are the result of normal breathing action, while smaller sub-millimeter deflections correspond to the individual’s heartbeat. While this research showed it was possible to detect vital signs acoustically from a distance, the paper says the ultrasound transmissions used were of a frequency and intensity not achievable by the consumer smart speaker hardware they were investigating.

In fact, the paper explains that standard smart speakers are not particularly well suited to ultrasonic transmission or reception in the first place, as they have naturally been designed to prioritize human speech and music. That said, the researchers reference the fact that Google’s Nest already uses ultrasonic range finding to adjust visual elements such as font size. Determining how far a viewer is standing from the screen is a far cry from picking up 0.5 mm vibrations in their carotid artery, but it’s a start.

The team’s sonar implementation was designed around the inherent limitations of smart speaker hardware, and generates a 50 ms frequency modulated continuous wave (FMCW) chirp between 18 kHz and 22 kHz. This allows the software to not only calculate a very precise distance to a stationary target, but ignore any echos that are returned beyond a predefined range.

Even still, the wavelength of sound at these frequencies is far larger than some of motions the software is looking for. With a bodily deflection of just 0.3 mm to 0.8 mm, individual heartbeats are particularly difficult to detect. Several layers of filters are applied to try and improve the signal-to-noise ratio, and adaptive beamforming allows the speakers to zero in on the target.

The paper concludes while that the resulting pulse data is not as accurate as the traditional electrocardiogram (ECG) readings being used as a control, it was close enough to be a useful diagnostic tool for both healthy patients and those suffering from various cardiac abnormalities. Especially when taking into account the advantages offered by the non-contact approach, particularly for telemedicine applications.

Ideal Conditions

While the researchers were able to pull some impressive data with their test setup, it should be said that there are several caveats which (at least for now) would make it difficult to put this technique into practice. For one thing, the smart speaker needs to be aimed at the patient’s chest and placed at a distance of 40 cm to 60 cm (15 in to 23 in). Accuracy drops off with distance, and at least in this version of the software, any returns beyond one meter are considered background noise and are ignored entirely.

Ideal placement for the smart speaker.

This is obviously at odds with normal smart speaker placement. Even if the speaker was close enough to the patient, say on a table next to a bed or chair, it’s normally going to be pointed up towards the ceiling. While testing did show the software was able to function when the patient was off to the left or right side, the error rate increased considerably.

To be useful, the device would likely need to be placed in a custom mount that would keep it in the ideal position while measurements were being taken. This isn’t a huge burden, and the cost would largely be negligible, but it does speak to the relative inflexibility of this approach.

Interestingly, the tests showed that the clothing worn by the patient had only a slight impact on the acoustic readings. While tight fitting garments produced the best results, even loose blouses and hospital gowns didn’t prevent the system from collecting useful data. Thick fabrics or multiple layers of clothing did attenuate the signal however, which could potentially be a problem depending on climate.

Similarly the paper found that the system couldn’t reliably extract a cardiac rhythm from extremely obese patients, as the excess tissue dampened the underlying bodily motion. Accordingly, patients with a body mass index (BMI) of 35 or higher were excluded from the primary test group. The software was separately evaluated with a group of five individuals who had a median BMI of 38.6, and of those, it was only able to detect the heart rate of one participant. The paper notes that this falls in line with the findings of previous experiments, and notes that obese patients simply aren’t good candidates for non-contact sensing technology at this time.

Breaking into The Walled Garden

Issues with speaker placement and the type of clothing the patient is wearing are, in the grand scheme of things, fairly minimal. Some refinement is obviously necessary, and clearly it won’t be as simple as just installing a piece of software on your Amazon Echo and getting the baby’s heart rate displayed on your phone without some special considerations being made, but all the pieces seem to be in place for somebody to run with the idea. Unfortunately, despite such promising research, bringing this capability to consumer smart speakers simply isn’t possible right now.

While the paper makes extensive references to these commercial devices, and uses their hardware capabilities as a limiting factor in all of their experiments, the researchers admit the software they’ve developed can’t actually be used on any currently available product from the likes of Google or Amazon. The simple fact is that the APIs available to developers don’t provide the low-level hardware access required to implement their sonar technique. Giving third party applications direct control over the speaker and microphones would raise obvious security concerns, so this type of functionality would either need to be built into the system from the start by the manufacturer, or potentially be relegated to specifically provisioned units intended for healthcare providers.

Taking The Long Way Around

Even if bureaucratic concerns keep the team’s software off of commercially available smart speakers, the research they’ve conducted is still extremely valuable. The possibilities offered by an affordable, automated method of taking a patient’s vitals without physical contact are undeniably compelling, especially with the lingering threat of COVID-19. Technology that can help monitor and diagnose potentially contagious patients without risking exposure to the nurses and physicians attending to them is in high demand, and will continue to be of great interest even when this current crisis is behind us.

miniDSP UMA-8-S

If commercial smart speakers aren’t a viable platform for this software, what then? Well for one thing, you could simply build your own hardware. As outlined at the end of the paper, all of the experiments were performed with a UMA-8-S seven microphone array from miniDSP and a PUI Audio AS05308AS-R speaker connected to desktop computer. The UMA-8-S is something of a development kit itself, and includes onboard digital signal processor (DSP) with beamforming capability. Marketed towards both professional and DIY users, it looks to be a particularly interesting device in its own right.

The team says the Python source code for their project is also available upon request, so long as you agree to a non-commercial license. Put simply, everything needed to recreate this project is readily available for anyone who wants it. Whether it becomes an area of exploration for biohackers or gets refined and marketed by a some tech entrepreneur, it seems inevitable that this technology will bubble its way to the surface in the coming years.

17 thoughts on “Hey Google, Is My Heart Still Beating?

  1. Impressive, yes. Practical real world benefits, yes.
    Potential for the already creepy world, of big tech spy technology, to become even more Orwellian, definitely. Reading your heart rate and monitoring your breathing, without touching the patient, has it’s obvious advantages in a hospital scenario, but I know, I for one, wouldn’t want that sort of thing in my house, especially with AI moving as fast as it has been.

  2. Very interesting. Props to the team for figuring this out. One has to wonder though, if they can implement heartbeat detection in the lab with only minor changes to the hardware, if this was part of the design. After all the privacy concerns raised with smart speakers, counting bodies remotely would just be another item on the list. 22 kHz is well beyond the audible range so the filtering must have been designed with some head room. while this sounds like a wild conspiracy and is mere speculation, the benefits this offers to law enforcement and advertisers seem obvious, so I wouldn’t rule it out as a possibility. Food for thought I guess…

    1. When these devices start talking to each other at 22khz so they know what other devices are in range. Sharing information, software updates, triangulation data for sniper targets on the room. Science fiction has a way of becoming science fact very quickly when it benefits those in the know.

  3. “The paper goes so far as to say that even with multiple people in the room, their technique can be used to monitor the heart and respiratory rate of a specific target individual.”

    Now just imagine a hostage situation with one of these speakers listening in. Stuff of science fiction.

    1. yes, but it seems like that quote is intentionally misleading to imply that it can single out any target individual in the room when I would infer from the rest of the text that it just means that it can measure the heartbeat of the person that it’s aimed at, while there’s other people making background noise.

      1. Read the paper. The researchers say they could ID multiple individuals by their breathing, and then use the beam forming to focus in on them selectively based on distance.

  4. Why not just make a cheap wearable measurement device instead?
    In the case of simple heart beat detection, the average smart watch has that now. I would assume that they’re not pulling much useful info beyond that, and seeing that it doesn’t actually work on current hardware that’s out there (without modification) it’s probably easier to do via a lot of other techniques. Laser is probably simpler than sonar if it needs to be non-contact.

  5. This is scary. It might have some use cases, like monitoring small children that sometimes suffers sudden death while asleep (or sth, i am not a doctor), but it is also scary as f***. Yet another reason NOT to have any “smart” nonsense in my house.

  6. Isn’t there a technique that employs positive feedback, where you can raise someone’s heart rate, based on synchronising with the current heartbeat?

    If this can be done subliminally, you could potentially induce a heart attack in the targeted individual, before they realise.

    “Attack of the killer Alexa”? Well, it would make a good Dr. Who episode, anyway.

  7. it says between 18kHz and 22 kHz, wikipedia says the common hearing range for humans is 20 to 20kHz, while some can reach 28kHz in laboratory conditions, so being close enough, you probably will hear it.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.