TapType: AI-Assisted Hand Motion Tracking Using Only Accelerometers

May 14, 2022 by Dave Rowntree 16 Comments

The team from the Sensing, Interaction & Perception Lab at ETH Zürich, Switzerland have come up with TapType, an interesting text input method that relies purely on a pair of wrist-worn devices, that sense acceleration values when the wearer types on any old surface. By feeding the acceleration values from a pair of sensors on each wrist into a Bayesian inference classification type neural network which in turn feeds a traditional probabilistic language model (predictive text, to you and I) the resulting text can be input at up to 19 WPM with 0.6% average error. Expert TapTypers report speeds of up to 25 WPM, which could be quite usable.

Details are a little scarce (it is a research project, after all) but the actual hardware seems simple enough, based around the Dialog DA14695 which is a nice Cortex M33 based Bluetooth Low Energy SoC. This is an interesting device in its own right, containing a “sensor node controller” block, that is capable of handling sensor devices connected to its interfaces, independant from the main CPU. The sensor device used is the Bosch BMA456 3-axis accelerometer, which is notable for its low power consumption of a mere 150 μA.

User’s can “type” on any convenient surface.

The wristband units themselves appear to be a combination of a main PCB hosting the BLE chip and supporting circuit, connected to a flex PCB with a pair of the accelerometer devices at each end. The assembly was then slipped into a flexible wristband, likely constructed from 3D printed TPU, but we’re just guessing really, as the progression from the first embedded platform to the wearable prototype is unclear.

What is clear is that the wristband itself is just a dumb data-streaming device, and all the clever processing is performed on the connected device. Training of the system (and subsequent selection of the most accurate classifier architecture) was performed by recording volunteers “typing” on an A3 sized keyboard image, with finger movements tracked with a motion tracking camera, whilst recording the acceleration data streams from both wrists. There are a few more details in the published paper for those interested in digging into this research a little deeper.

The eagle-eyed may remember something similar from last year, from the same team, which correlated bone-conduction sensing with VR type hand tracking to generate input events inside a VR environment.

Continue reading “TapType: AI-Assisted Hand Motion Tracking Using Only Accelerometers” →

No Privacy: Cloning The AirTag

February 22, 2022 by Al Williams 55 Comments

You’ve probably heard of the infamous rule 34, but we’d like to propose a new rule — call it rule 35: Anything that can be used for nefarious purposes will be, even if you can’t think of how at the moment. Case in point: apparently there has been an uptick in people using AirTags to do bad things. People have used them to stalk people or to tag cars so they can be found later and stolen. According to [Fabian Bräunlein], Apple’s responses to this don’t consider cases where clones or modified AirTags are in play. To prove the point, he built a clone that bypasses the current protection features and used it to track a willing experimental subject for 5 days with no notifications.

According to the post, Apple says that AirTags have serial numbers and beep when they have not been around their host Apple device for a certain period. [Fabian] points out that clone tags don’t have serial numbers and may also not have speakers. There is apparently a thriving market, too, for genuine tags that have been modified to remove their speakers. [Fabian’s] clone uses an ESP32 with no speaker and no serial number.

The other protection, according to Apple, is that if they note an AirTag moving with you over some period of time without the owner, you get a notification. In other words, if your iPhone sees your own tag repeatedly, that’s fine. It also doesn’t mind seeing someone else’s tags if they are near you. But if your phone sees a tag many times and the owner isn’t around, you get a notification. That way, you can help identify random tags, but you’ll know if someone is trying to track you. [Fabian] gets around that by cycling between 2,000 pre-loaded public keys so that the tracked person’s device doesn’t realize that it is seeing the same tag over and over. Even Apple’s Android app that scans for trackers is vulnerable to this strategy.

Even for folks who aren’t particularly privacy minded, it’s pretty clear a worldwide network of mass-market devices that allow almost anyone to be tracked is a problem. But what’s the solution? Even the better strategies employed by AirGuard won’t catch everything, as [Fabian] explains.

This isn’t the first time we’ve had a look at privacy concerns around AirTags. Of course, it is always possible to build a tracker. But it is hard to get the worldwide network of Bluetooth listeners that Apple has.

Make Your Own BLE-Enabled IOS App From Scratch

December 4, 2021 by Orlando Hoilett 20 Comments

Even those readers who are most skeptical of Apple products will like this Bluetooth Low Energy (BLE)-enabled iOS app tutorial from [Akio].

With everything being “connected” these days smartphone applications are of course a ubiquitous part of our existence. We’ve seen plenty of examples connecting your Bluetooth-enabled projects to an Android device, but comparatively fewer tutorials for connecting to iOS devices. This mostly has to do with Android’s much larger market share and also Android’s more open-source friendly business model. Nevertheless, if you do much IoT development either as a hobby or professionally, then you probably find yourself interacting with Apple devices more than you like to admit.

[Akio’s] app is essentially updating a chart, in real-time, with data read from an Adafruit nRF52832 Feather board. He then walks you through all the basics of creating a user interface (UI) using Apple’s Storyboard interface, a simple drag-and-drop scheme similar to something you’ve probably used in many other contexts. [Akio] shows readers how to add buttons for allowing users to interact with the app, labels for displaying data to the user, as well as walks you through Apple’s odd methodology of connecting UI elements to code using IBAction and IBOutlets. The highlight of his tutorial is showing readers how to add charts to their iOS apps which seems to take a few more steps than you might imagine.

[Akio] does a really good job detailing all the relevant functions so that readers will hopefully understand what each piece of the code is doing. And we really enjoyed him adding individual video tutorials for some of the trickier programming steps. He also readily admits that some folks may opt to develop their UI exclusively in code as opposed to the Storyboard but he argues that the Storyboard is still important for beginners and is really handy when the UI is fairly simple.

Of course, in true open-source fashion, [Akio] provides all his code on his GitHub repository so you can clone the repo and run the code yourself as well as credit some of the resources he used while making his app. Two things we really love to see. Hopefully, [Akio’s] tutorial will make connecting to iOS devices seem much less onerous than it once was.

Turn On Your Lights With A Wave Of A Magic Wand

April 22, 2021 by Lewin Day 14 Comments

Smartphones and voice assistants are the typical way most of us interact with our smart devices around the home, but it doesn’t have to be the only way. [Sam March] wanted things to feel a little more magical – so built a wand to do the job instead.

The wand relies on a DA14531 Bluetooth Low Energy (BLE) system-on-chip, and is paired with what appear to be smart plugs running on the same hardware. With an accelerometer in the wand, it’s able to detect waving motions, and then signal the smartplugs over Bluetooth to switch outlets on or off. As far as the magic side of things is concerned, [Sam] took his lead from [Arthur C. Clarke], who famously stated “Any sufficiently advanced technology is indistinguishable from magic.” Thus, efforts were made to miniaturize the electronics down to a single tiny PCB, allowing it to be secreted inside a turned wooden wand that’s wrapped in leather.

The end result is a fun project that’s also probably useful when [Sam] wants to turn the lights off without getting out of bed. We could imagine that, configured properly to work on a room-by-room basis, it could be useful for guests who don’t know where the light switches are.

If the name sounds familiar, it’s because we’ve heard from [Sam] before – with his great DIY smartwatch build. Video after the break.

Continue reading “Turn On Your Lights With A Wave Of A Magic Wand” →

A Crash Course On Sniffing Bluetooth Low Energy

March 23, 2021 by Tom Nardi 25 Comments

Bluetooth Low Energy (BLE) is everywhere these days. If you fire up a scanner on your phone and walk around the neighborhood, we’d be willing to bet you’d pick up dozens if not hundreds of devices. By extension, from fitness bands to light bulbs, it’s equally likely that you’re going to want to talk to some of these BLE gadgets at some point. But how?

Well, watching this three part video series from [Stuart Patterson] would be a good start. He covers how to get a cheap nRF52480 BLE dongle configured for sniffing, pulling the packets out of the air with Wireshark, and perhaps most crucially, how to duplicate the commands coming from a device’s companion application on the ESP32.

The first video in the series is focused on getting a Windows box setup for BLE sniffing, so readers who aren’t currently living under Microsoft’s boot heel may want to skip ahead to the second installment. That’s where things really start heating up, as [Stuart] demonstrates how you can intercept commands being sent to the target device.

It’s worth noting that little attempt is made to actually decode what the commands mean. In this particular application, it’s enough to simply replay the commands using the ESP32’s BLE hardware, which is explained in the third video. Obviously this technique might not work on more advanced devices, but it should still give you a solid base to work from.

In the end, [Stuart] takes an LED lamp that could only be controlled with a smartphone application and turns it into something he can talk to on his own terms. Once the ESP32 can send commands to the lamp, it only takes a bit more code to spin up a web interface or REST API so you can control the device from your computer or other gadget on the network. While naturally the finer points will differ, this same overall workflow should allow you to get control of whatever BLE gizmo you’ve got your eye on.

Continue reading “A Crash Course On Sniffing Bluetooth Low Energy” →

Get Apple To Track Your Bluetooth Devices For You

March 5, 2021 by Tom Nardi 10 Comments

Apple’s “Find My” service allows users to track their missing devices by leveraging a worldwide network of location-aware iGadgets. With millions of iPhones and Macs out in the wild listening for the missing device’s Bluetooth advertisements and relaying their findings to the Cupertino Mothership, it’s a highly effective way of tracking hardware so long as it stays in relatively urban areas. Unfortunately, the system is completely proprietary and non-Apple devices aren’t invited to play.

Or at least, that used to be the case. A project recently released by the [Secure Mobile Networking Lab] called OpenHaystack demonstrates how generic devices can utilize Apple’s Find My network by mimicking the appropriate Bluetooth Low Energy (BLE) broadcasts. Currently they have a firmware image for the BBC micro:bit, as well as a Python script for Linux, that will allow you to spin up an impromptu Find My target. But the team has also published all the information required to implement similar functionality on other BLE-capable devices and microcontrollers, so expect the list of supported hardware to grow shortly.

Somewhat ironically, while OpenHaystack allows you to track non-Apple devices on the Find Me tracking network, you will need a Mac computer to actually see where your device is. The team’s software requires a computer running macOS 11 (Big Sur) to run, and judging by the fact it integrates with Apple Mail to pull the tracking data through a private API, we’re going to assume this isn’t something that can easily be recreated in a platform-agnostic way. Beyond the occasional Hackintosh that might sneak in there, it looks like Tim Cook might have the last laugh after all.

It’s not immediately clear how difficult it will be for Apple to close this loophole, but the talk of utilizing a private API makes us think there might be a built-in time limit on how long this project will be viable. After all, Big Tech doesn’t generally approve of us peons poking around inside their machinations for long. Though even if Apple finds a way to block OpenHaystack, it’s expected the company will be releasing “AirTags” sometime this year which will allow users to track whatever objects they like through the system.

Cycling Cadence Display With ESP32

March 2, 2021 by Danie Conradie 9 Comments

Terry Pratchett once said “Wisdom comes from experience. Experience is often a result of lack of wisdom.” This is as true with technical skills as it is with the rest of life, and you won’t truly understand a specific topic unless you’ve struggled with it a bit. [publidave] wanted a simple wireless display for a bluetooth cycling cadence sensor, and soon found himself deep down the rabbit hole of Micropython and Bluetooth Low Energy on the ESP32.

[publidave] had converted his bicycle for indoor training during lockdown and winter, and realized he can’t use the guided training app and view his cadence simultaneously, so he needed a dedicated cadence display. Since [publidave] was comfortable with Python, he decided to give Micropython on the ESP32 ago. Bluetooth Low Energy can be rather confusing if you haven’t implemented it before, especially if good examples are hard to come by. In short, the ESP32 needs to find the sensor, connect to it, select the right service, and listen for the notifications containing the data. The data is then converted to RPM and displayed on a small OLED display. [publidave] does an excellent job of describing what exactly he did, highlighting the problems he encountered, and how he solved them.

In the end, he had a functional display, a good idea of what he would do differently next time, and a lot of additional knowledge and understanding. In our book that’s a successful project.

Since so much of the health related devices work with Bluetooth Low Energy, it could be handy to know the technology and how to interface with it. It would allow you to do things like unbrick a $2000 exercise bike,