University of Washington researchers studying the potential medical use of smart speakers such as Amazon’s Echo and Google’s Nest have recently released a paper detailing their experiments with non-contact acoustic heartbeat detection. Thanks to their sensitive microphone arrays, normally used to help localize voice commands from the user, the team proposes these affordable and increasingly popular smart home gadgets could lead a double life as unobtrusive life sign monitors. The paper goes so far as to say that even with multiple people in the room, their technique can be used to monitor the heart and respiratory rate of a specific target individual.

Those are some bold claims, but they aren’t without precedent. Previous studies performed at UW in 2019 demonstrated how smart speaker technology could be used to detect cardiac arrest and monitor infant breathing. This latest paper could be seen as the culmination of those earlier experiments: a single piece of software that could not just monitor the vitals of nearby patients, but actually detect a medical emergency. The lifesaving potential of such a program, especially for the very young and elderly, would be incredible.

So when will you be able to install a heart monitor skill on the cheap Echo Dot you picked up on Prime Day? Well, as is often the case with this kind of research, putting the technique to work in the real-world isn’t nearly as easy as in the laboratory. While the concept is promising and is more than worthy of further research, it may be some time before our lowly smart speakers are capable of Star Trek style life-sign detection.

Can You Hear Me Now?

The researchers explain in the paper that their work is built on earlier experiments that used powerful ultrasound transducers to detect minute displacements of a patient’s chest, abdomen, and neck. Relatively large motions are the result of normal breathing action, while smaller sub-millimeter deflections correspond to the individual’s heartbeat. While this research showed it was possible to detect vital signs acoustically from a distance, the paper says the ultrasound transmissions used were of a frequency and intensity not achievable by the consumer smart speaker hardware they were investigating.

In fact, the paper explains that standard smart speakers are not particularly well suited to ultrasonic transmission or reception in the first place, as they have naturally been designed to prioritize human speech and music. That said, the researchers reference the fact that Google’s Nest already uses ultrasonic range finding to adjust visual elements such as font size. Determining how far a viewer is standing from the screen is a far cry from picking up 0.5 mm vibrations in their carotid artery, but it’s a start.

The team’s sonar implementation was designed around the inherent limitations of smart speaker hardware, and generates a 50 ms frequency modulated continuous wave (FMCW) chirp between 18 kHz and 22 kHz. This allows the software to not only calculate a very precise distance to a stationary target, but ignore any echos that are returned beyond a predefined range.

Even still, the wavelength of sound at these frequencies is far larger than some of motions the software is looking for. With a bodily deflection of just 0.3 mm to 0.8 mm, individual heartbeats are particularly difficult to detect. Several layers of filters are applied to try and improve the signal-to-noise ratio, and adaptive beamforming allows the speakers to zero in on the target.

The paper concludes while that the resulting pulse data is not as accurate as the traditional electrocardiogram (ECG) readings being used as a control, it was close enough to be a useful diagnostic tool for both healthy patients and those suffering from various cardiac abnormalities. Especially when taking into account the advantages offered by the non-contact approach, particularly for telemedicine applications.

Ideal Conditions

While the researchers were able to pull some impressive data with their test setup, it should be said that there are several caveats which (at least for now) would make it difficult to put this technique into practice. For one thing, the smart speaker needs to be aimed at the patient’s chest and placed at a distance of 40 cm to 60 cm (15 in to 23 in). Accuracy drops off with distance, and at least in this version of the software, any returns beyond one meter are considered background noise and are ignored entirely.

This is obviously at odds with normal smart speaker placement. Even if the speaker was close enough to the patient, say on a table next to a bed or chair, it’s normally going to be pointed up towards the ceiling. While testing did show the software was able to function when the patient was off to the left or right side, the error rate increased considerably.

To be useful, the device would likely need to be placed in a custom mount that would keep it in the ideal position while measurements were being taken. This isn’t a huge burden, and the cost would largely be negligible, but it does speak to the relative inflexibility of this approach.

Interestingly, the tests showed that the clothing worn by the patient had only a slight impact on the acoustic readings. While tight fitting garments produced the best results, even loose blouses and hospital gowns didn’t prevent the system from collecting useful data. Thick fabrics or multiple layers of clothing did attenuate the signal however, which could potentially be a problem depending on climate.

Similarly the paper found that the system couldn’t reliably extract a cardiac rhythm from extremely obese patients, as the excess tissue dampened the underlying bodily motion. Accordingly, patients with a body mass index (BMI) of 35 or higher were excluded from the primary test group. The software was separately evaluated with a group of five individuals who had a median BMI of 38.6, and of those, it was only able to detect the heart rate of one participant. The paper notes that this falls in line with the findings of previous experiments, and notes that obese patients simply aren’t good candidates for non-contact sensing technology at this time.

Breaking into The Walled Garden

Issues with speaker placement and the type of clothing the patient is wearing are, in the grand scheme of things, fairly minimal. Some refinement is obviously necessary, and clearly it won’t be as simple as just installing a piece of software on your Amazon Echo and getting the baby’s heart rate displayed on your phone without some special considerations being made, but all the pieces seem to be in place for somebody to run with the idea. Unfortunately, despite such promising research, bringing this capability to consumer smart speakers simply isn’t possible right now.

While the paper makes extensive references to these commercial devices, and uses their hardware capabilities as a limiting factor in all of their experiments, the researchers admit the software they’ve developed can’t actually be used on any currently available product from the likes of Google or Amazon. The simple fact is that the APIs available to developers don’t provide the low-level hardware access required to implement their sonar technique. Giving third party applications direct control over the speaker and microphones would raise obvious security concerns, so this type of functionality would either need to be built into the system from the start by the manufacturer, or potentially be relegated to specifically provisioned units intended for healthcare providers.

Taking The Long Way Around

Even if bureaucratic concerns keep the team’s software off of commercially available smart speakers, the research they’ve conducted is still extremely valuable. The possibilities offered by an affordable, automated method of taking a patient’s vitals without physical contact are undeniably compelling, especially with the lingering threat of COVID-19. Technology that can help monitor and diagnose potentially contagious patients without risking exposure to the nurses and physicians attending to them is in high demand, and will continue to be of great interest even when this current crisis is behind us.

If commercial smart speakers aren’t a viable platform for this software, what then? Well for one thing, you could simply build your own hardware. As outlined at the end of the paper, all of the experiments were performed with a UMA-8-S seven microphone array from miniDSP and a PUI Audio AS05308AS-R speaker connected to desktop computer. The UMA-8-S is something of a development kit itself, and includes onboard digital signal processor (DSP) with beamforming capability. Marketed towards both professional and DIY users, it looks to be a particularly interesting device in its own right.

The team says the Python source code for their project is also available upon request, so long as you agree to a non-commercial license. Put simply, everything needed to recreate this project is readily available for anyone who wants it. Whether it becomes an area of exploration for biohackers or gets refined and marketed by a some tech entrepreneur, it seems inevitable that this technology will bubble its way to the surface in the coming years.