Shoulder Surfing With OpenCV


While it seems that many people are wise to shoulder surfing, keeping a lookout for anyone spying on their passwords, [Haroon] wrote in to remind us that the threat is just as real today as it ever was.

The subjects of his research are touch screen phones and tablets, which utilize on-screen keyboards for data entry. He says that while nearly all password entry boxes on these devices are obscured with the traditional line of asterisks, the keyboards themselves are quite an interesting vulnerability.

Since touch screen technology can be finicky at times, most vendors ship their devices with some sort of key press verification system. On the iPhone and iPad, for instance, each key is highlighted in blue following a button press. This functionality makes it quite easy for shoulder surfers to casually steal your password if you’re not paying attention.

But what if you are well aware of your surroundings? [Haroon] has developed a piece of software he calls shoulderPad, which is based on openCV that does the surfing for him. The application can monitor a video stream, live or recorded, extracting the user’s password from the highlighted button presses. His demonstrations show the recording taking place at a relatively close distance, but he says that it would be quite easy to use surveillance footage or zoom lenses to capture key presses from afar.

He does say that the button highlighting can be easily disabled in the iPhone’s options pane, which should negate this sort of attack for the most part.

Continue reading to see a quick video of shoulderPad in action.


14 thoughts on “Shoulder Surfing With OpenCV

  1. It would be nice if it was disabled only for password fields. Since disabling it globally is unnecessary and possibly annoying.

    Or maybe it does that?

    And then there is the fact that phones and tablets show each character you type in the text box momentarily before obscuring it. You could scan that for passwords too in a video stream of sufficient resolution (but obviously the keyboard reading is easier to see).

  2. Why there is no google+ button here;)?

    as for this “hack” – great job, and certainly is not only applicable to blue hilighting keyboards – just that simple color change made it easier to do.

    If we can see the password box, the finger position and when letters appear, we can take finger postion and assume that button under it was pressed at the time. Well, now you can troll your acquaintances saying that Android is more secure because there is so much disparity in keyboard apps and display sizes;)

    1. Hi Josh.

      It is indeed the stock keyboard, but with “complex passwords” enabled.

      The white keyboard shows up for regular typing, and this one shows up for password prompts.

  3. I think you could steal anybody’s password from any keyboard (SW or HW) if you get to film them as they do it (and you know the layout of the keyboard, or can discern it in the footage). I really see no protection against that, unless we really start typing in passwords in Morse code…

  4. @Anonymous: I’m with you on this one. A randomised keyboard layout for password fields (different each time) would not only negate shoulder surfing, but also prevent build up of fingerprints in certain positions on the screen, which, while it may not be such a big deal for occasionally used online passwords, definitely is an issue for some unlock codes.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.