Stay Smarter Than Your Smart Speaker

Smart speakers have always posed a risk to privacy and security — that’s just the price we pay for getting instant answers to life’s urgent and not-so-urgent questions the moment they arise. But it seems that many owners of the 76 million or so smart speakers on the active install list have yet to wake up to the reality that this particular trick of technology requires a microphone that’s always listening. Always. Listening.

With so much of the world’s workforce now working from home due to the global SARS-CoV-2 pandemic, smart speakers have suddenly become a big risk for business, too — especially those where confidential conversations are as common and crucial as coffee.

Imagine the legions of lawyers out there, suddenly thrust from behind their solid-wood doors and forced to set up ramshackle sub rosa sanctuaries in their homes to discuss private matters with their equally out-of-sorts clients. How many of them don’t realize that their smart speaker bristles with invisible thorns, and is even vulnerable to threats outside the house? Given the recent study showing that smart speakers can and do activate accidentally up to 19 times per day, the prevalence of the consumer-constructed surveillance state looms like a huge crisis of confidentiality.

So what are the best practices of confidential work in earshot of these audio-triggered gadgets?

Image by Roberto Parada via the Atlantic

Hey Alexa, Who’s to Blame?

On the one hand, people and their tastes are the problem. That sleek cylinder or stubby hockey puck of an assistant is designed to be attractive enough that people will display it prominently in their homes. Most people don’t want their appliances to come with constant reminders of their inherent dangers, because warnings are visual noise.

Take stoves for instance. Stoves designers assume that their users have a certain level of base knowledge. It goes with out saying that hot stoves are hot, but you weren’t born knowing that — you either learn that by burning your hand, or by heeding the warnings of other humans.

With smart speakers, the downsides are far less obvious, and the injury less instant. It’s easy to be drawn in by their technological siren song, because a faceless assistant with a pleasing voice that exists to serve and can provide most any answer immediately is an exciting and attractive idea. Come up with something like that, and you can pretty much write the book on how it looks. At least, until public opinion or litigation makes you change your mind.

xkcd #1807

Privacy Protection Possibilities

Should the makers of smart speakers shoulder some of the responsibility of saving us from ourselves? Probably. Power tools have warnings and dead man switches, but that’s because misusing power tools has immediate, obvious, and painful consequences. Misappropriated data can have life-altering consequences, too. Just ask anyone who’s had their life turned upside down because of a geofence warrant.

In seeking out an answer to this issue, more control seems like a good start. Physical solutions are the safest way, second only to not having an open mic in the first place. Most smart speakers have a mic mute button, but you’ll have to remember later why Alexa’s not answering you.

An easy way to kill the network connection would be nice, too. You already have the power to isolate your smart speaker(s) on their own network and give it the ol’ man-in-the-middle finger whenever you need to — but that requires basic network admin skills that many people don’t have. A software dashboard would be useful for disabling them for chunks of time without leaving the couch, so consider setting up your work hours as times when Siri, Cortana, and Okay Google are put into a cone of silence.

Of course, smart speakers aren’t the only ubiquitous products with microphones, so solving the consumer-created problems are just the start. The best solution to all of this nonsense may be to usher in an era of protective fashion, starting with bracelets that drown microphones in ultrasonic waves.

51 thoughts on “Stay Smarter Than Your Smart Speaker

  1. One simple solution to this whole problem is. “Don’t use smart speakers/assistants.”

    Another more tricky solution is to roll your own where the server is in your hands. Then just have it go and grab the data for you. Here you would have all the logs yourself, and you can always trim it to do what you want it to do. Though, obviously with the added hassle of actually implementing the whole thing and make it work….

    I myself on the other hand doesn’t use smart assistants at all… Mostly since I always have a computer in reach regardless…
    Then there is also the fact that smart assistants are rarely even able to be of help, nor understand what one is after. (Considering that Google can only do a half bad job at answering technical questions… Not that one should search in the way to start with… Answering a single question can require multiple searches for different pieces of info that together form the answer.)

    1. “Roll your own,” doesn’t apply here. In the millions of Americans out there who own these items lie a small percentage of above average aptitude for electronics. This isn’t a doobie. It requires a lot of time to develop the infrastructure. There are ‘guides’ that can show you the at home DIY version primarily aimed at the few who possess the tools and general understanding. I could build one of these things–a lot of us here could develop some iteration loosely resembling a smart speaker. But I that take a of time, not necessarily in hardware. It’s the *software*. I prefer to use a solution at a very low price point, and I’ll just be mindful that Google is listening to me talk about how hard it is to find 5.56mm right now.

      1. “Though, obviously with the added hassle of actually implementing the whole thing and make it work….”
        And there I weren’t even referring to the hardware.

        Yes, the vast majority of people wouldn’t even have a clue where to even start. Let alone know how to work with the data and formulate it into useful search quires to prod at the internet with, then there is the API’s for interfacing with search engines, and reading off results from web pages, etc. Filter useful info from things like ads, etc, while not having the assistant being long winded and blurb out everything…..

        Yes, it is obviously very far from trivial.

        But that doesn’t change the fact that one can technically still do it.
        And if people were to try to roll their own, then who knows, maybe someone figures out a good way to do it.

        In terms of hardware, well, a raspberry pi, a USB microphone or two, and then one just needs a speaker…. Doesn’t have to be pretty to be functional.

    2. What gets me, I can go into any major electronic store and get hardware that will connect to Google or Amazon, but a simple voice recognition hardware in an item that I train directly and does not need the internet access is not available.

      I build my own voice recognition hardware/software and it is not that hard, however without the powerful computers/databases of the big boys the system that I train with my voice do not work with anyone else in the house.

      I see that as a PLUS.

      I should be able to buy a smart xxxxx and quickly train it will a word list, result a smart item that does not leak my data outside the home.

      Voice control of speakers, lights, garage doors, burglar alarms … etc should not need to tell anyone else what want to get the job done. Plus no spying on what I am saying or disabling functions/items or even worse depending on a server that goes off-line and leaves me with worthless junk.

  2. Regarding confidential/classified info, we have all learned in recent years that as long as there’s no intent, it’s ok to break the law. Or maybe that only applies to certain people, I can’t remember.

    But seriously. I don’t have always-on listening devices in my home and have Siri turned off. But it does make you wonder about attack vectors…

    1. Well, you should have known better than to break the law, but no reasonable prosecutor will bring your charges to court, as long as the president has a chat about his grandkids with her. Personally, I assume that everything I say is going to be recorded and used against me. That being given, since there’s no 4th and 5th–and arguably 3rd–amendment protections, why would a reasonable person cooperate at all with the law? The 3rd amendment prohibits quartering of soldiers in your home. The underlying reason is to suppress political activity, which the 4th and 5th are additionally there to protect and ensure that all prosecutions are legal and not punitive. I would say, allowing 3rd parties to report your information to the government should fall under the 3rd amendment because they then, remotely, become quartered agents of the government in your home. Fruit of the poison tree.

  3. I wish a hacker would made something like a “don’t listen device” for smart speakers, something like a man in the middle device disconected from internet that sits between the microphones and the smart speaker main board; that device would be listening 24/7 and would let the audio pass to the board if a keyword is said before the regular voice command; and would also cut the mics with a timer or another keyword. That would require some speech recognition in a really small device, but since it only would need to recognize one or two keywords I think It can be done.
    An external RGB led would be a good indicator in order to know if the smart speaker can listen you or not.

        1. when I saw that project I had an even better idea…. have it “whisper” nonsense words to screw up their advertisement profile. Even better get enough people using these that you can sell product placement “whispers” to companies where the device will say your company or product’s name x times per day, ostensibly driving that product up the list for that company’s “most talked about” item leading to more prominent placement in google/amazon search results….

    1. I’ve been looking into this, mostly because I don’t want to recite a corporate mantra every time I want to actuate a switch.

      There’s already a few implementations, but I don’t think there’s any that don’t require some invasive work.

      1. Looks great but I don’t think that thing could completely isolate the built in mics of most smart speakers. But if we remove the built in mics of the smart speaker and wire that device to where the mic were, It should do the job.

    2. That’s basically what these home assistant devices already do.
      They have a small self contained always awake sub-system that is always listening for the wake word, running a rolling buffer locally but not uploading.
      When the wake work is detected, it wakes the main part of the device up and feeds the audio of the wake word, plus a bit more (before and after) to account for the warm boot time of the main system and make sire none of the question or command is missed, which when booted then takes over to do the rest of the work until the thing is done and it goes back to sleep.

      Barring false positives, the home assistant isn’t uploading audio all the time.

      1. That should be the regular working method, but It doesn’t work like that, since If you start a conversation about anything near a smartphone or a smart speaker and then you surf the internet, you’ll find ads related to your last conversation (It doesn’t matter if you’ve never talked or searched about the topic of your conversation before).
        I’ve tried speaking of bikes (I dislike them so I never speak of them), of diapers (don’t have small children), of exotic fruits (never talked about exotic fruits before) and boooooom! ads or recomendations of related apps a little while after the conversation.

        1. I was observing the same thing and found something even scarier that might or might not be a connected issue: a former class mate of my wife who is now the CEO of a mid-size publishing house came by our place to drop off some documents when he came through our town one day. He has never before been at our place and we didn’t mention his company when he handed us the documents. One hour after he left, I was bombarded with Instagram ads about this very publishing house for about two days. This has happened never before nor ever since. We don’t have any smart devices and I’m the only one using Siri (and only in “passive” mode)… our iPhones were the only devices in the room technically capable of “listening” and even though “off” they apparently did exactly that…

          1. I think it more likely that there was something in your online search history that triggered this activity. I will certainly try this with my iPhone and other devices, though, to see if there is any suspicious activity.

          2. even the general public believes that our phones are always-on microphones being listened to and interpreted by robots, because that’s the easiest concept to grasp mentally.

            now i won’t try and say that our devices aren’t recording audio, because they probably are occasionally. however, when you have spooky experiences like the one you encountered i think there are deeper mechanisms at play.

            google also most likely perform quiet exchanges between people’s phones when they are geographically equal to one another. your CEO friend’s phone is probably already full of publishing-related searches and e-mails, and my suspicion is that phones sort of talk to each other when they’re in the same “room” so to speak. as disconcerting as it is that they might be listening, it may not be literally audio monitoring, but a kind of cross-pollination between users’ interests and histories that interact in proximity.

            i have no facts to back this up, by the way. it’s totally a theory, but i think google’s algorithm assumes people that close to each other might be friends and your ads will serve up things that they like, in attempt to expand your ad horizons.

        2. I’ve tried that too. I even talked about odd things AFTER I activated Alexa. I asked Alexa about baby bottles and child car seats. I see no indication of my strange interest in items anywhere when I am browsing online. I don’t see things in Amazon either. I can’t attest for Google or any other smart speakers, but I can’t ever get Amazon Echo/Alexa to indicate anything I talk about … not showing up as ads online, no emails, nothing. I’m going to call “fake news” on this one.

          1. With most people here having a background in science / tech, the term “fake news” should be reserved to behaviour where sb is obviously spreading false information. If however, sb is reporting the observation of a real phenomenon and leaves it open what mechanism / intention might be tied to this observed phenomenon, a more constructive response would be “needs to be validated through systematic experimentation” or “could be coincidence”. This way this forum will continue to be a place for a productive dialog between tech minded people.

  4. I’ve sometimes worried about whether smartphones have a similar vulnerability. Although so far I haven’t gotten paranoid enough to try lifting traces and installing physical on/off switches on my phone’s microphone and GPS devices.

      1. And google assistant, which samsung won’t allow to be removed from my J7 “unlocked” (PT: it’s bootloader is locked, the unlocked only refers to carrier…disappointment ensues).

    1. Why the paranoia? I won’t ask to divulge any confidential information here in public but I find a lot of people fear-monger over privacy concerns. What are you concerned about getting out? Statistically speaking, you aren’t likely to be targeted by anyone at all unless you hold some high position of power or carry a hefty wallet. Hackers spend their time looking for good victims that they would risk prison over, not just some concerned citizen with too many artificially sought after moon rocks in their collection.

      1. It’s not hackers that are after you. It is organized crime! The data they collect is brokered to special interest groups who’d like to know in advance which houses in a target neighbourhood are most quiet, carry a high insurance premium and have interior photo’s shared on Facebook. Or they lift just enough personal information to take out a substantial loan in your name. The possibilities to slice collated data are endless, and unless you are the perpetrator, you are the victim.

      2. Hackers (perhaps better to use the word crackers to avoid confusion with the tinkere type of hacker when talking on this site) traget the low hanging fruit, regardless of how much they can gain from that low hanging fruit. Malware campaigns, with a few exceptions, go for who-ever has a vulnerable internet device. When abusing a neo-stasi alexa unit becomes something which can be fully automated, frauds carried out by doing so will be carried out where-evr possible, not solely against high value targets. Think regular spam and phishing, not spear-phishing and ceo-whaling, when crooks automate they don’t need to focus on targets any more. That is what these little gestapo boxes could facilitate, automated attack tools to commandeer them, automated recognition of financial or sensitive discussions, automated thefts from bank accounts or automated blackmail messages.

  5. Once upon a time you could use something like snips. Snips wouldn’t spew all your personal information to amazon! I wish somebody created something similar to replace it..

    1. I am not familiar with Snips (snips.AI?), but there’s Mycroft. I haven’t tried it yet, and i don’t know if it’s exactly the same, but i plan on trying it at some point.

    2. I’m quite familiar with “snips” betetr known as wire cutters or pliers, you take the alexa’s power cord out of the socket, then apply snips to it. After than no more neo-stasi in your house.

  6. I recently had to replace my cell phone, costing me a whopping $7.49 for a decent LG smartphone (Tracfone is great!). One of the first things I did while perusing the manual (strange thing for a guy to do, but I’m weird anyway) was to try out the voice control feature. “That’s cool,” I thought as I disabled the feature moments later.
    This isn’t Star Trek. Having a computer listen to your every word and report it back to Google, Amazon, or (shudder) Apple is NOT A GOOD THING.

  7. I speculated to friends a while back that the Oval Office Ukraine whistleblower was simply an Echo Dot. Counsel refused to identify because:

    a. they didn’t have to
    b. too embarrased

  8. “A software dashboard would be useful for disabling them for chunks of time without leaving the couch, so consider setting up your work hours as times when Siri, Cortana, and Okay Google are put into a cone of silence.”

    El cheapo: use a timer on the outlet your smart device(s) is attached. Off during work hours. On afterwards. Reasonable?

  9. There is a TV ad in Oz that shows a google phone user sending a picture to a friend using voice control. Every time it plays, our google speaker responds with I can’t do that because you don’t have any photos!

  10. There’s two sides to every coin; this technology can also be used to prove innocence. If it’s able to log everything you say and your location at given times, it can rescue you from false accusations, which happen more than you think! I had an incident where the police attempted to charge me with a hit-and-run, but I was in another state at the time. I was able to prove my innocence through electronically time-stamped photos and receipts. The police appeared more annoyed than thankful, which meant they had start at the beginning again, having to do some real detective work as opposed to just busting anyone who fits the description!

  11. False triggers are a problem in and of themselves, beyond any leaking of vital household trivia. My wife’s a psychologist / counsellor and having a smart speaker activating and trying to respond to words while a client is hypnotised is a problem!

    I suggested to Google that a mode where the speaker needed a tap to activate would be great. The rep seemed to like the idea and promised to pass it onto the devs. I’m not holding my breath for an implementation…

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.