Camera software learns to pick you out of a crowd

tld_object_tracking

While the Kinect is great at tracking gross body movements and discerning what part of a person’s skeleton is moving in front of the camera, the device most definitely has its shortfalls. For instance, facial recognition is quite limited, and we’re guessing that it couldn’t easily track an individual’s eye throughout the room.

No, for tracking like that, you would need something far more robust. Under the guidance of [Krystian Mikolajczyk and Jiri Matas], PhD student [Zdenek Kalal] has been working on a piece of software called TLD, which has some pretty amazing capabilities. The software uses almost any computer-connected camera to simultaneously Track an object, Learn its appearance, and Detect the object whenever it appears in the video stream. The software is so effective as you can see in the video below, that it has been dubbed “Predator”.

Once he has chosen an object within the camera’s field of vision, the software monitors that object, learning more and more about how it looks under different conditions. The software’s learning abilities allow it to pick out individual facial features, follow moving objects in video, and can recognize an individual’s face amid a collection of others.

While the software can currently only track one object at a time, we imagine that with some additional development and computing horsepower, this technology will become even more amazing.

Comments

  1. blaine says:

    imagine how this could help in the tracking of suspected terrorists. with it tied into networked cameras in airports world wide. wonder if it can learn from scanned photos of the same subject?

  2. Jesse says:

    yeah, or protesters and dissidents! Use something like this to track them from a protest to their cars or homes.

  3. Just me says:

    Scary.

  4. Andy Goth says:

    Tracking gross body movements? Maybe this can be marketed to gastroenterologists. ;^)

  5. Anon Ymous says:

    There are no military uses for this technology at all. Nope, none.

  6. Nick says:

    the csiro in Australia actually developed a similar system several years ago to monitor for terrorists and alert authorities when a suspect is captured by a surveillance camera

  7. Evan Reeves says:

    Echoing some of the privacy issues above, I wonder how this does when you slap on the Groucho Marx glasses and mustache?

  8. Stevie says:

    Wow that system is insanely powerful! Good job, that’s gonna be one rich dude

  9. Doktor Jeep says:

    I can’t help but notice that my fellow Americans don’t come up with stuff like this, or that they don’t share it.

    But I expect there are Americans developing software like this and it will probably be shooting at me someday.

  10. Japa says:

    This is very interesting indeed, though pulling in a second person would have been helpful for demonstrating it.

  11. wardy says:

    Uncanny. That’s some military spec coding. Something like this will be winning the next DARPA desert challenge, I’d wager.

  12. ds2ktj says:

    Anyone know if this runs under Octave instead of Matlab? I’m not a student and really don’t want to cough up the $$$ to play with this.

  13. John says:

    …”But I expect there are Americans developing software like this and it will probably be shooting at me someday.”

    LOL! :)

    As far as the s/w goes, it’s pretty awesome. I just hope the development reaches the point that it is not reliant on Matlab soon…

  14. G says:

    If anyone thinks that systems like this aren’t ALREADY being used for surveillance, they’re kidding themselves. I had image processing and recognition classes in college back in ’92. Processing is way faster now but the principles have been around for a while.

    And, it makes no sense to compare this to Kinect. People need to remember that the Kinect is just RGB video and depth… all the skeleton tracking etc. is done in SOFTWARE on the xbox. Here’s an idea, marry this software WITH a Kinect and see what you can do. The Kinect is hardware, this is software, we’re talking classic apples and oranges.

  15. woutervddn says:

    if this could read eye-irises.. you’d have a darn good terrorist spotter.. :O but as is, I think military goals aren’t the main use for this. (Although this could make a fairly cheap and easy landing system for UAVs that are to small to fit a radar in..)

  16. yrss says:

    Cool video. Bad article.

    Feature tracking is old hat, enough so that limited implementations are a common student project at this point. The cool thing here is the way he’s applying it to HCI.

  17. EvilEngineer says:

    Well, it works on the small scale. But how does the software handle a database of hundreds of thousands or even millions of objects to check references for?

    Facial recognition databases are huge, it would need some serious storage and processing power to quickly recognize a tracked object in a very large data set.

  18. Very cool. Just waiting for OSX install instructions now, and I’ll give it a whirl!

    I think you’d need a very high res camera for this to be useful at somewhere like an airport. So many people come and go that there are bound to be false positives – but I suppose it could be worked on!

    Anyway – Good work.

  19. kalleguld says:

    @EvilEngineer: Good point, i’m guessing it won’t scale very well. But for checking of public places, you could connect it to a larger non-realtime database, that looks up unrecognized faces. And then let this piece of sw work on faces seen in the last 10 mins to track detailed behavior.

    And yes, this has military/surveillance/oppressive applications, but it’s not a job for science to evaluate that – leave that to humanists :)

  20. Jay says:

    I could have SWORN I already saw this here on Hackaday recently, but I probably saw it somewhere else. The method he uses isn’t as complicated as some people seem to think… I tossed together something similar in about an hour after seeing this for the first time. (I wanted to try the “learning” aspect of the project since I never had)

  21. John says:

    Hi Jay,
    Any chance your code is open source?

  22. Jay says:

    @John

    Nope, but I’d gladly explain the concepts involved. If you (using your preferred lang) know how to capture webcam frames and read the color data from the image, you can have this basic thing up and running in no time.

  23. John says:

    Jay,

    I’m sure we’d all be ecstatic to get a rundown of the concepts if you’re willing. You have the stage, or let me/us know your preferred forum for your presentation.
    Thanks for your insight!

  24. Reventlov says:

    Very difficult to install.
    Matlab is a piece of shit, octave isn’t yet supported, and we need gcc 4.3.
    Wtf ?

  25. Rob says:

    An uncle of mine (Patric G Sobalvarro, if you care to look at some of the other badassery he has created) has a company that for several years has been providing an extremely advanced software package that does everything this webcam hack does, and more, and does it in crowds across multiple cameras.

  26. LJ says:

    Very interesting and I’ll be playing with this in some depth at some point in the future.

    A comparison between some of the other tracking methods would have been useful in the papers (I just skimmed, may well have missed it if it was there). The great thing about it is that this is OS thus hopefully can be used to complement some of the tracking methods used in the frameworks that are out there.

  27. Jay says:

    @John (and anyone else interested)

    I tossed up instructions to get you going on an object tracker. I hope it helps! :D
    http://www.kaldobsky.com/tracking/

  28. John says:

    @Jay

    Much appreciated, Sir! This ought to keep us busy with something to hack up over the weekend. :)

  29. rusty says:

    dangerous is the right word to describe this.

  30. Bob says:

    DL’ed both the compiled version and the source code. So far, the compiled version works great but the matlab source code is being a pain. I would recommend giving it a DL and try.

  31. Climatebabes says:

    This smart naive guy just solved a problem for a bunch of older, overagressive, paranoid phantasy assasins.

    The technology is not revolutionary, but it will sure help the police and other non-democratic forces like Xe and homeland security keep an eye on us.

    Young people should 100% boycott science, as it stands science only leads to destruction.

  32. john says:

    Facial recognition apps: Fear and Loathing in Las Vegas

    Rob from Washington writes that he’s very interested in new cloud-enhanced facial recognition apps that can be used to pick terrorists out of crowds. You know, like the sweaty mob of 120,000 wild-eyed nerds being herded through CES this week. It seems that, using the cloud, they now are able to precisely match up 140 different points on a face, even if that face is beneath a full beard or a Clone Wars Trooper’s helmet.
    I’ll try to sneak into the invitation-only Homeland Security contractor session and find out about that. If I’m never heard from again, you’ll know what happened. They recognized my face, in spite of the shades…

    Oh, and if I’m not detained by security, I’ll let you know about the shiny new phones and tablets at CES, too…
    January 3rd, 2011

  33. Jay says:

    Wow, I just noticed that 122 unique visitors checked out my object tracker page! I thought, at most, I’d get a handful of you guys… not 122! With those kinds of numbers I guess I can sit back and wait for a bunch of new vision-based projects to be posted by the hackaday community. :)

  34. Gert says:

    “Hello John Anderton”

    Minority Report.

  35. jswanson says:

    @ds2ktj…I am still chasing down some errors with Octave/Debian, but it looks like it will be possible. Hoping to get something up and running next week. Awesome idea to say the least.

  36. Maave says:

    Slap this on a 3DS to track eye distance and separation, then match the stereoscopic image accordingly.

  37. infrared says:

    lame… object tracking is nothing new but getting it to work reliably on a wide variety of camera sensors is kinda cool. I have not been able to get a response from the authors of the matlab code. But anyone with decent high level OO programming can do this with a few functions.

  38. Thanh says:

    @Jay, thanks very much for the write up of the algorithm of your program. I’ve read and thought about trying it out. However, I have question on this problem: how to search for the object when it’s moved out of the frame and then moved back?
    I read your text saying using stored pvalues to search for previous match. But I’m not sure how to do this. Any info is much appreciated.

    @Bob, I tried running the pre-compiled binary program. It had an error saying it could not find version 7.13 of the MCR. I don’t have matlab. Is there a free library for Matlab that I can install to get around this? Thanks

    -Thanh

  39. Cram says:

    I foresee future warfare conversations:

    “BOOM HEADSHOT!”

    “Stop hacking! Stupid aimbot.”

  40. Dennis says:

    I can see someone using this to ID Parts.
    Against a set of online parts suppliers.

  41. tdbone1 says:

    “Could not find version 7.13 of the MCR”

  42. Tobs says:

    Woah! This software is scaring, eventhough it’s quite impressive.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 96,576 other followers