Winning Game-App Contests with Computer Vision


[Gadget Addict] found out about a contest being held by a shoe seller. Their mobile app has a game very much like Bejeweled. The high scorer each month gets £500. His choices were to be better at the game than everyone else, or to be smarter. He chose the latter by writing a computer vision program to play the game.

There are two distinct parts of a hack like this one. The first is just figuring out a way to programmatically detect the game board and correctly identify each icon on it. This is an iPad game. [Gadget Addict] is mirroring the screen on his laptop, which gives him easy access to the game board and also allows for simulated swipes for automatic play. Above you can see two examples where black pixels may be counted in order to identify the icon. A set of secondary checks differentiates similar entries after the first filtering. The other part of the hack involves writing the algorithms to solve for the best move.

If you liked this one, check out a super-fast Bejeweled solver from several years back. We should also mention that this was just a proof of concept and [GA] never actually entered the contest.

Bejeweled Blitz bot makes your high score look just sad

Programmers don’t need to get good at a game to achieve a high score, they code a bot for that instead. Take [hypnotizd] for instance. He was learning to write in the C# language and decided to make a bot that plays Bejeweled Blitz on Facebook. He figures he took between 48 and 72 hours of coding over a couple of weeks, but remember, he was learning the language at the same time. We think you’d be hard pressed to achieve a 1.5 million range score by yourself, even with that amount of practice time.

We spoke with [hypnotizd] yesterday afternoon to get a bit of background on how he made this happen. His code (he’s not releasing it so you’ll have to write your own) scrapes the screen image as input. You can see at the beginning of the video after the break that he sizes his app to properly align each jewel in its grid. The program then identifies each game piece by finding the center of the cell and taking a 25 square-pixel average color. Many of the jewels are easily recognized in this first pass, but some are harder and require several different tests to identify. That’s the difficult part, choosing the best move is just a matter of coming up with your own rules on how the bot should play the game.

[Read more...]


Get every new post delivered to your Inbox.

Join 96,705 other followers