Perspective tracking with only a web cam

[Johhny chung Lee], eat your heart out. Check out what these guys are doing with face tracking and immersive 3d as their final project in class. They’re using a singe camera and an FPGA to produce the demo you see in the video. Facial tracking is done by skin color, so that might have some issues in some environments, but being able to have perspective shift with you, without rigging up some more hardware is fantastic.

We realize that this is completely different that what [Johnny] is doing. We love [Johnny]‘s work and think it is ground breaking to be able to pull this stuff off with a cheap game controller. We just couldn’t help but draw the parallel from the end result.

[thanks Bruce]


  1. andrew says:

    OpenCV has face tracking in the example code section. I wonder why they didn’t use that.

  2. djrussell says:

    1. the guy in the back needs to put some pants on. too much leg.

    2. ever play a video game and really get into enough to be leaning and moving around as if you could look around objects? this could actually make it work and would be a really cool in game effect.

  3. toodlestech says:

    My webcam came stock with face tracking drivers and physically moves if I start to go out of range of its sight… for $40.

  4. pascal says:

    @andrew it looks like the prime reason for this project is the FPGA, they seem to process the webcam image and generate the output image using only that prototype board. On a GPU it would require 50 lines of OpenGL/CV-Code, but that’s really not the point.

  5. pascal says:

    (also I think the “without rigging up some hardware”-Part is hilarious, since their solution *is* pure hardware^^)

  6. Chris says:

    > OpenCV has face tracking in the example code section. I wonder why they didn’t use that.

    fun? or because it was for a class that probably wasnt about using OpenCV example code.

    >My webcam came stock with face tracking drivers and physically moves if I start to go out of range of its sight… for $40.

    not nearly as cool as building it yourself on FPGA though huh? Since when is hack a day full of old curmudgeons that think its better to just buy the completed project instead of doing it yourself?

  7. Chuck says:

    @andrew We did consider using OpenCV’s face tracking function for our project. But we didn’t know how to use OpenCV library with the Nios II soft processor that we implemented on the FPGA.

  8. yoyoyo says:

    artificial mirror

  9. wally says:

    @andrew: I did exactly that for part of my dissertation at Essex Uni. It didn’t take much to implement it …

  10. Snowy says:

    Either that guy was mumbling or my hearing is going. With all the background noise and him not looking at the camera when talking/mumbling, I couldn’t understand a word he said.

  11. Kaepora says:

    I can imagine great and fun immersive video games where each individual limb can be tracked independently. This is a great start.

    (I saw this guy in the lab the other day…leaning around and looking at a bunch of squares… I know what he was doing.)

  12. wally says:

    @andrew: That said, having looked over this guy’s videos a second time, they look much smoother than I managed. Kudos to them :)

  13. sean says:

    Okay, I don’t mean to hate, but….
    what he’s doing is so completely basic that it isn’t complicated at all. If you download processing, one of the examples associated with web cam programming is almost this exact same thing…
    All this guy is doing is putting a dot in the center of the brightest object detected by the camera. Sure, this COULD be used as perspective, but technically perspective has to do with the space in which you can see. All this is doing is tracking your head’s placement… not in which direction you are looking. True perspective software would focus on your eyes.
    Now I agree that this opens up a lot of possibilities for games and etc., but I don’t think that what this guy has done should be a huge deal.

    Otherwise, good find.

  14. walt says:

    that dude in the background needs to put on some shoes and socks. what is with all the youtube videos of people in their bare feet? GROSS!

  15. anon says:

    it is human feet. get over it.

  16. Yaniv says:

    A similar project back from 2008, using openCV and Johnny Lee’s software:

  17. Ahmed Sokar says:

    Nice work

    May you guide me to run OpenCv on NiosII

  18. samir says:

    Hello I need also your help on running opencv on nios II

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

Join 96,597 other followers