Winning Game-App Contests With Computer Vision

[Gadget Addict] found out about a contest being held by a shoe seller. Their mobile app has a game very much like Bejeweled. The high scorer each month gets £500. His choices were to be better at the game than everyone else, or to be smarter. He chose the latter by writing a computer vision program to play the game.

There are two distinct parts of a hack like this one. The first is just figuring out a way to programmatically detect the game board and correctly identify each icon on it. This is an iPad game. [Gadget Addict] is mirroring the screen on his laptop, which gives him easy access to the game board and also allows for simulated swipes for automatic play. Above you can see two examples where black pixels may be counted in order to identify the icon. A set of secondary checks differentiates similar entries after the first filtering. The other part of the hack involves writing the algorithms to solve for the best move.

If you liked this one, check out a super-fast Bejeweled solver from several years back. We should also mention that this was just a proof of concept and [GA] never actually entered the contest.

10 thoughts on “Winning Game-App Contests With Computer Vision

  1. This is a good example of KISS – Keep is simple stupid!

    Instead of using a complicated opencv algo, he’s just stripping it back to basics and counting pixels.

    It looks like the game forces you to wait for all animations to finish before you can make another move, slows things down a lot!

  2. I was expecting to see an arduino with a servo touching slowly and inaccurately at the screen. I’m glad he had the sense to go the software route instead of the ‘oh look I used an arduino’ route.

    Whats the best way to mirror the ipad to a computer anyway? Looks pretty smooth in the video.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.