Wearables are ubiquitous in today’s society. Such devices have evolved in their capabilities from step counters to devices that measure calories burnt, sleep, and heart rate. It’s pretty common to meet people using a wearable or two to track their fitness goals. However, a big question remains unanswered. How accurate are these wearable devices? Researchers from the Big Ideas Lab evaluated a group of wearables to assess their accuracy in measuring heart rate.
Unlike other studies with similar intentions, the Big Ideas Lab specifically wanted to address whether skin color had an effect on the accuracy of the heart rate measurements, and an FDA-cleared Bittium Faros 180 electrocardiogram was used as the benchmark. Overall, the researchers found that there was no difference in accuracy across skin tones, meaning that the same wearable will measure heart rate on a darker skin-toned individual the same as it would on a lighter skin-toned. Phew!
However, that may be the only good news for those wanting to use their wearable to accurately monitor their heart rate. The researchers found the overall accuracy of the devices relative to ECG was a bit variable with average errors of 7.2 beats per minute (BPM) in the consumer-grade wearables and 13.9 BPM in the research-grade wearables at rest. During activity, errors in the consumer-grade wearables climbed to an average of 10.2 BPM and 15.9 in the research-grade wearables. It’s interesting to see that the research-grade devices actually performed worse than the consumer devices.
And there’s a silver lining if you’re an Apple user. The Apple Watch performed consistently better than all other devices with mean errors between 4-5 BPM during rest and during activity, unless you’re breathing deeply, which threw the Apple for a loop.
So, it seems as if wrist-worn heart rate monitors still have some work to do where accuracy is concerned. Although skin tone isn’t a worry, they all become less accurate when the subject is moving around.
If you’d like to try your own hand with fitness trackers, have a look at this completely open project, or go for the gold standard with a wearable DIY ECG.
This isn’t too surprising. The only evidence for the claim that wrist-worn optical heart rate sensors don’t work properly in people with darker skin was a non-statistically-significant difference in one research paper and a Reddit comment by one black guy who may well have just been wearing his wrong. Seriously, that was the entire basis for those articles.
There was, however, one demographic group that the paper in question did find these heart rate monitors worked less well in: men. Only statistically significant result they got, if I remember rightly, and substantially larger than the difference due to skin colour. This didn’t produce a wave of headlines because obviously the device manufacturers weren’t secretly ignoring the main customers for these things. (It seems to be an inherent limitation of the tech, something to do with wrist diameter most likely.)
It’s not that surprising that the consumer-grade devices did well either. Companies have put a lot of resources into trying to have the best, most accurate, most reliable heart rate monitoring on their devices.
I think it’s disingenuous to say that skin color is only brought up because of a reddit comment. I was a design engineer at a pulse oximeter manufacturer and dark pigmentation definitely needs to be considered in development. If you introduce a light blocking pigment, you’ll introduce a DC offset and have less returning signal to work with. In the earlier days of pulse oximeters, this was a significant problem. Since the wearables are often designed by people with no medical device background, and since turning down the LEDs will conserve energy, allowing a longer runtime, it’s reasonable to suggest that a wearable that hadn’t considered skin pigmentation could have problems, and this should be tested.
That said, an AVERAGE error of 13.9 BPM at rest? Given that a resting heart rate outside the range of 60 to 100 bpm would be bradycardic or tachycardic, I could guess that your resting heart rate is 80 bpm, and i’d have a max error of around 20, and an average error less than that. I don’t even need LEDs or detectors to build this wearable.
Given that I’ve seen heart rate watches “detect” a pulse when sitting on a white piece of paper on the desktop, I’d be inclined to think that lighter skin would be more likely to have higher errors. Speculation thus far is that it’s picking up some harmonic of AC lighting frequency or something. I have not tried the same test on a purely sunlit desktop.
They definitely pick up AC lighting frequencies, but a good device will attempt to minimize this primarily by sampling at 75 Hz, which helps to partially cancel both 50 and 60 Hz lighting flicker. If you were sure it would only ever be used in a specific country, you could sample just at 50 Hz or 60 Hz and really minimize the (short term) effect.
Would have been interesting to also include consumer rings like the Oura Ring and I think there are another one or two.
We’re actually looking into those rings for a research project. What we found is the biometric rings measure very intermittently and only at rest or during sleep. I’m not sure there is a software mode to get continuous data making them hard to evaluate a test like this. They may only provide average heart rate over several hours (or minutes) making it difficult to compare those devices to an electrocardiogram (ECG) that gives beat-to-beat heart rate. I guess you could average the ECG over the same time scale as the ring. That could probably work, I guess.
Thank you yes by default it is taking snapshots and comparing against a benchmark. Apparently it does read during the day but not instant exercise type readings, and it is defaulting to doing its stats at night. It is focussed more on overall body health, and not exercise I’ve found out.
Okay cool. That’s about the same information we gathered as well. I’ll spill the beans a bit. The research group I work with is trying our hand at a biometric ring. I can share more details when it’s published.
Looking forward to seeing that – II get alerts daily from your RSS feed so should not miss it.
Can it show my heart bit BBB condition?
“It’s interesting to see that the research-grade devices actually performed worse than the consumer devices.”
I think this has a lot to do with the development of highly optimized Pulse Oximetry and Rate integrated sensors used in mass produced consumer grade inexpensive finger-tip Pulse Oximeters.
I agree with you for sure. Also, I think that Apple and Fitbit have had much more time to refine their algorithms than some of the companies that produce research-grade devices. Another point to mention is the research-grade devices are really intended to provide researchers with hardware so they can build their own algorithms not necessarily relying solely on the algorithms that are already running on the device.