Augmented Reality With An FPGA




[Julie Wang] has created an augmented reality system on a Field Programmable Gate Array (FPGA). Augmented reality is nothing new – heck, these days even your tablet can do it. [Julie] has taken a slightly different approach though. She’s not using a processor at all. Her entire system, from capture, to image processing, to VGA signal output, is all instantiated in a FPGA.

Using the system is as simple as holding up a green square of cardboard. Viewing the world through an old camcorder, [Julie’s] project detects and tracks the green square. It then adds a 3D image of Cornell’s McGraw Tower on top of the green. The tower moves with the cardboard, appearing to be there. [Julie] injected a bit of humor into the project through the option of substituting the tower for an image of her professor, [Bruce Land].

[Julie] started with an NTSC video signal. The video is captured by a DE2-115 board with an Altera Cyclone IV FPGA. Once the signal was inside the FPGA, [Julie’s] code performs a median filter. A color detector finds an area of green pixels which are passed to a corner follower and corner median filter. The tower or Bruce images are loaded from ROM and overlaid on the video stream, which is then output via VGA.

The amazing part is that there is no microprocessor involved in any of the processing. Logic and state machines control the show. Great work [Julie], we hope [Bruce] gives you an A!

25 thoughts on “Augmented Reality With An FPGA

  1. What would happen if you went outside into the green world? Crazy!
    Chroma key picks the colour least likely to be in the unnatural world.
    FPGA, the closest thing yet on the brain.

    1. Could chroma key be improved, by using a something switches between two states in short intervals to identify itself so that it may be used between things that even include it’s own colour? I’m thinking of a solution based on a LCD panel that switches between black and white or maybe some sort of IR beacon solution, any ideas?

      1. Probably an easy way to do it is to have a tablet flash colors on the screen. Overkill but just about everyone already owns a device that would work, maybe even one (or more) that would otherwise be unused.

    2. QR code or QR code style patterns seems to be a popular solution for augmented reality.
      I am not sure how difficult it will be, it might even be easier.
      But this is excellent as a prove of concept and senior project.

    1. As a woman I just want to point out that the reason why we get irked when people are passing a complement, is because those comments don’t exist in a vacuum.

      When it turns into a pattern of people commenting on your appearance, it just reinforces the idea that we’re judged to varying degrees on our appearance.

      I say this because some men seem genuinely confused when women react negatively to what men think are harmless comments. It’s a “death by a thousand paper cuts” sort of situation.

      I guess it’s also kind of weird? I don’t feel this urge to comment on the appearance of men as often as some men do.

      Hackaday has a serious problem if I can look at the opening of an article and make solid predictions about what I’m going to find in the comments.

  2. Hacking and making is such a wonderfully open community, where ideas and knowledge are shared without prejudice, for the simple joys of learning and understanding. It’s really a remarkable community, and often represents the very best of what the human race is capable of.

    And then there’s your comment. Pathetic, narrow minded racism. Real bottom of the barrel stuff. This has no place in any community, especially one devoted to sharing ideas.

    If you were trying to be funny, you not only failed but also managed to be offensive and repulsive. If this is a genuinely held belief, then please feel free to FOAD. Your views are not welcome in the hacker community.

  3. Interesting, I learned a new word today.

    As a European, I am always surprised how racist the US is. I did not learn about the word “wetback” until I went out with a mexican born professor in the US, who was heckled from across the street.

      1. The comment was posted while I was asleep. The first time a number of users clicked report, and it popped out to moderation. The person posted it again, and I nuked it myself. We really do try to avoid censoring things here, but trolls will be trolls. Apologies to [Julie] that the comment was here in the first place.

  4. I always try to at least skim this site, no matter how tired or rushed I may be.
    I saw and wondered if that remark was a sly use of wording related to the article to get in a ugly thought
    Or was i just missing some “tech jargon”
    As one of the proponents of Hackadays light handed moderation
    I don’t envy the staffs task of choosing when it’s time to remove
    one of our missives.
    may I say that I do wish there was a practical way to leave something of a placeholder to make it easier to follow the flow of what happened
    in the comments section.
    Perhaps a plain block with the routine “comment removed”
    and the offending name intact for the rest of us to see who
    was trying to ruin things for everyone else?
    It’s almost funny how I was wondering if we’d see some sophomoric
    comments about the camera angle…..
    ( a fellows never to old for hormones ,ehh?)
    But instead it goes this way.

    – – –
    p.s. I hope the “Shatnerizer” option hasn’t mangled my post once again.

      1. Well, someone needs to call it. You guys are procrastinating on this “update comment system” item :)

        We have seen lots of updates (the project site, the mega prize etc) recently but not this. I’ve been around for the last few years and I see this come up regularly.

        I understand the job is delicate but you gotta start somewhere.

  5. Awesome project, at first I was thinking: Why would someone want to do that with a FPGA?, as it is much easier to do with a microprocessor.
    But then I realized that it was BECAUSE it is difficult she made this choice.
    The easy path seldom leads to greatness.
    If Julie doesn’t already has a job waiting for her (she is about to graduate soon if I understand things correctly), I do not think she has to worry about that for very long.

  6. A microprocessor is also logic and state machines, however already connected. It is possible to put microprocessor’s net into FPGA creating a very expensive variant of the processor (FPGA prices are much higher than simple microprocessors/micro controllers — actually this is the reason we have them — they are much more economical to use). Therefore, I miss a little bit the purpose of this achievement. Looks like a student exercise, luckily the FPGA can be easily reprogrammed to do the stuff that general purpose microprocessor can’t.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.