Camera Software Learns To Pick You Out Of A Crowd


While the Kinect is great at tracking gross body movements and discerning what part of a person’s skeleton is moving in front of the camera, the device most definitely has its shortfalls. For instance, facial recognition is quite limited, and we’re guessing that it couldn’t easily track an individual’s eye throughout the room.

No, for tracking like that, you would need something far more robust. Under the guidance of [Krystian Mikolajczyk and Jiri Matas], PhD student [Zdenek Kalal] has been working on a piece of software called TLD, which has some pretty amazing capabilities. The software uses almost any computer-connected camera to simultaneously Track an object, Learn its appearance, and Detect the object whenever it appears in the video stream. The software is so effective as you can see in the video below, that it has been dubbed “Predator”.

Once he has chosen an object within the camera’s field of vision, the software monitors that object, learning more and more about how it looks under different conditions. The software’s learning abilities allow it to pick out individual facial features, follow moving objects in video, and can recognize an individual’s face amid a collection of others.

While the software can currently only track one object at a time, we imagine that with some additional development and computing horsepower, this technology will become even more amazing.


42 thoughts on “Camera Software Learns To Pick You Out Of A Crowd

  1. imagine how this could help in the tracking of suspected terrorists. with it tied into networked cameras in airports world wide. wonder if it can learn from scanned photos of the same subject?

  2. the csiro in Australia actually developed a similar system several years ago to monitor for terrorists and alert authorities when a suspect is captured by a surveillance camera

  3. I can’t help but notice that my fellow Americans don’t come up with stuff like this, or that they don’t share it.

    But I expect there are Americans developing software like this and it will probably be shooting at me someday.

  4. …”But I expect there are Americans developing software like this and it will probably be shooting at me someday.”

    LOL! :)

    As far as the s/w goes, it’s pretty awesome. I just hope the development reaches the point that it is not reliant on Matlab soon…

  5. If anyone thinks that systems like this aren’t ALREADY being used for surveillance, they’re kidding themselves. I had image processing and recognition classes in college back in ’92. Processing is way faster now but the principles have been around for a while.

    And, it makes no sense to compare this to Kinect. People need to remember that the Kinect is just RGB video and depth… all the skeleton tracking etc. is done in SOFTWARE on the xbox. Here’s an idea, marry this software WITH a Kinect and see what you can do. The Kinect is hardware, this is software, we’re talking classic apples and oranges.

  6. if this could read eye-irises.. you’d have a darn good terrorist spotter.. :O but as is, I think military goals aren’t the main use for this. (Although this could make a fairly cheap and easy landing system for UAVs that are to small to fit a radar in..)

  7. Cool video. Bad article.

    Feature tracking is old hat, enough so that limited implementations are a common student project at this point. The cool thing here is the way he’s applying it to HCI.

  8. Well, it works on the small scale. But how does the software handle a database of hundreds of thousands or even millions of objects to check references for?

    Facial recognition databases are huge, it would need some serious storage and processing power to quickly recognize a tracked object in a very large data set.

  9. Very cool. Just waiting for OSX install instructions now, and I’ll give it a whirl!

    I think you’d need a very high res camera for this to be useful at somewhere like an airport. So many people come and go that there are bound to be false positives – but I suppose it could be worked on!

    Anyway – Good work.

  10. @EvilEngineer: Good point, i’m guessing it won’t scale very well. But for checking of public places, you could connect it to a larger non-realtime database, that looks up unrecognized faces. And then let this piece of sw work on faces seen in the last 10 mins to track detailed behavior.

    And yes, this has military/surveillance/oppressive applications, but it’s not a job for science to evaluate that – leave that to humanists :)

  11. I could have SWORN I already saw this here on Hackaday recently, but I probably saw it somewhere else. The method he uses isn’t as complicated as some people seem to think… I tossed together something similar in about an hour after seeing this for the first time. (I wanted to try the “learning” aspect of the project since I never had)

  12. @John

    Nope, but I’d gladly explain the concepts involved. If you (using your preferred lang) know how to capture webcam frames and read the color data from the image, you can have this basic thing up and running in no time.

  13. Jay,

    I’m sure we’d all be ecstatic to get a rundown of the concepts if you’re willing. You have the stage, or let me/us know your preferred forum for your presentation.
    Thanks for your insight!

  14. An uncle of mine (Patric G Sobalvarro, if you care to look at some of the other badassery he has created) has a company that for several years has been providing an extremely advanced software package that does everything this webcam hack does, and more, and does it in crowds across multiple cameras.

  15. Very interesting and I’ll be playing with this in some depth at some point in the future.

    A comparison between some of the other tracking methods would have been useful in the papers (I just skimmed, may well have missed it if it was there). The great thing about it is that this is OS thus hopefully can be used to complement some of the tracking methods used in the frameworks that are out there.

  16. DL’ed both the compiled version and the source code. So far, the compiled version works great but the matlab source code is being a pain. I would recommend giving it a DL and try.

  17. This smart naive guy just solved a problem for a bunch of older, overagressive, paranoid phantasy assasins.

    The technology is not revolutionary, but it will sure help the police and other non-democratic forces like Xe and homeland security keep an eye on us.

    Young people should 100% boycott science, as it stands science only leads to destruction.

  18. Facial recognition apps: Fear and Loathing in Las Vegas

    Rob from Washington writes that he’s very interested in new cloud-enhanced facial recognition apps that can be used to pick terrorists out of crowds. You know, like the sweaty mob of 120,000 wild-eyed nerds being herded through CES this week. It seems that, using the cloud, they now are able to precisely match up 140 different points on a face, even if that face is beneath a full beard or a Clone Wars Trooper’s helmet.
    I’ll try to sneak into the invitation-only Homeland Security contractor session and find out about that. If I’m never heard from again, you’ll know what happened. They recognized my face, in spite of the shades…

    Oh, and if I’m not detained by security, I’ll let you know about the shiny new phones and tablets at CES, too…
    January 3rd, 2011

  19. Wow, I just noticed that 122 unique visitors checked out my object tracker page! I thought, at most, I’d get a handful of you guys… not 122! With those kinds of numbers I guess I can sit back and wait for a bunch of new vision-based projects to be posted by the hackaday community. :)

  20. lame… object tracking is nothing new but getting it to work reliably on a wide variety of camera sensors is kinda cool. I have not been able to get a response from the authors of the matlab code. But anyone with decent high level OO programming can do this with a few functions.

  21. @Jay, thanks very much for the write up of the algorithm of your program. I’ve read and thought about trying it out. However, I have question on this problem: how to search for the object when it’s moved out of the frame and then moved back?
    I read your text saying using stored pvalues to search for previous match. But I’m not sure how to do this. Any info is much appreciated.

    @Bob, I tried running the pre-compiled binary program. It had an error saying it could not find version 7.13 of the MCR. I don’t have matlab. Is there a free library for Matlab that I can install to get around this? Thanks


Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.