Projection Mapping In Motion Amazes

Projection mapping is pretty magical; done well, it’s absolutely miraculous when the facade of a building starts popping out abstract geometric objects, or crumbles in front of our very eyes. “Dynamic projection mapping onto deforming non-rigid surface” takes it to the next level. (Watch the video below.)

A group in the Ishikawa Watanabe lab at the University of Tokyo has a technique where they cover the target with a number of dots in an ink that is only visible in the infra-red. A high-speed (1000 FPS!) camera and some very fast image processing then work out not only how the surface is deforming, but which surface it is. This enables them to swap out pieces of paper and get the projections onto them in real time.

Make sure that you do watch the video. The bit where they aim a fan at the projection-mapped T-shirt, and the dancing animated Stanford Bunny flaps in the breeze along with it is truly amazing.

Want more of the projection-mapping craze of the early 2010’s? Check out this post on projection mapping with industrial robot arms, or a projection mapped sandbox.  If that’s not enough for you, have a look at this footage from [Amon Tobin]’s live tour in 2012.

Thanks [Stuart] for the tip!

23 thoughts on “Projection Mapping In Motion Amazes

    1. I couldn’t agree more, awesome! But where do you do projection mapping in feature films? Do you mean for backgrounds behind actors? I’m a big behind-the-scenes video watcher so that’s very interesting to me.

  1. ok, despite the facts that it is cool… what is the practical application. Clothing…sure that would be fun, but the projector and camera system tracking the shirt, they need to be somewhere. So in other words, who will be using this technology and in what context, despite from looking awesome and cool, the video told me nothing about that. Could someone explain?

    1. We use projectors and foam board at at our nightclub as a way of making the DJ console look a bit more interesting for the different acts that we get in. It is possible something like this could make setting up a lot easier.

    2. Theatre!! Projection mapping in theatre is one of the largest emerging fields. At current it is used on stationary sets, or items being flown on automated winching systems, that produce the same motion every time. The possibility of projecting onto costume/props/human moved set opens another range of creative possibilities.

    3. As opposed to a specific application, let’s take a moment to think about this fundamentally; projection enables a non-display surface to become a display. Look around your room, your house, your office. Think about how you could use something that changes those things into displays. This is the part of the push into Augmented Reality technologies.

      The high speed mapping and correction for non-rigid basically allows real-world deployments (as opposed to carefully controlled ones). They mention fashion; the one that pops to mind is a white t-shirt or pants onto which they project different patterns and designs. Instant prototype shirt! How about theater? Like Pepper’s Ghost, you can have someone in a white suit and have a virtual someone else projected onto them. Or advertising; you have a UAV blimp cruising around a defined space and now it’s a jumbotron without the hassle of hanging something.

      Again, those examples are possible in AR if everyone observing is using the same glasses, where-as project enables everyone to experience the AR without wearing anything.

        1. Could have gone the OTHER way with that and talked about AR Porn ;^)

          For what it’s worth, I can see the tech moving beyond the IR dots. Around 1990 my step-father worked at a high-end workstation back (Silicon Graphics?) that had an optical mouse, however it only worked on a special laser etched metal pad. Today, optical mice have little problem tracking the various scratches and dirt on a normal surface. While I won’t purpose a specific solution, I see no reason a future system couldn’t initialize/calibrate/track via another method.

          1. When I first saw the post, I wondered if they were doing optical flow like you’re suggesting. With the resolution that they’ve got, that would be incredible — I hesitate to say “impossible” these days.

            An optical mouse tracks one point — the mouse’s center — by comparing differences over a 4×4 (usually) matrix, and updating pretty fast. To keep track of, say 640×480 locations with 16x640x480 values would be computationally challenging with today’s tech. (Unless there’s some clever math in there that I can’t rule out either.)

            The IR dots is relatively less impressive. Except that to capture motion accurately they need such a high speed camera and rapid processing, which in itself is a technical feat that would be impossible five or ten years ago for less than a million bucks…

            So in sum, this work is _merely_ very impressive, not absolutely magical. But it’s awesome regardless.

  2. What really makes this possible is the 1000fps projector that university has. Data acquisition speeds are much more easily achieved than those of the output. The biggest limiting factor in mapping projected objects is the output frequency as your output is compared to directly to the physical object, which is difficult to map as things like delay, jitter, smoothness, accuracy are all much more obvious. Updating more frequently has obvious demands, but reduces the amount of computation for prediction, which usually results in the under/over-compensation typical in automation tracking. Lowering the time between draws allows the tracked object to be more accurately mapped as errors become much less obvious.

    1. I had thought the VISION system was 1000fps, not the projector, but your comment had me returning to the video I see that they do indeed have a 1000fps PROJECTION system. I wonder how the system would fair with slower projection speeds such as 120fps.

      Also, isn’t this the second video from this university / team on Hackaday, where the first one was a 1000fps ball tracking system that used galvometers? (Maybe I saw that elsewhere…)

      1. They actually have many projects with this projector as the center piece and they’re all great. They’re pushing the research forward! As to 120fps they would 8.3ms+ to deal with, but in reality I think the best they could do would be ~25ms round trip, meaning they’d need to predict that far into the future ( various accel curves etc ); 1000fps is just ludicrous, even looking at a modest 6frame round trip we’re talking about 6ms, which now I need to go check their documentation as I bet they’ve measured this. It’s a beautiful thing!

  3. The first thing I would’ve done is my own version of D.A.N.C.E. by Justice….. save me a lot of post editing.

    And because of Halloween, some sort of digital disection for kicks…

    Oooohhh… a digital operation game? Holy crap… all I can think of is how I can turn this into entertainment.

  4. I wonder if 4 of these miniaturised projector are placed around a object such as a white tank in the desert, and the 4 projection modules have coresponding cameras viewing the scenery around the tank, the background could be projected on to the tanks making them stealthy and almost invisible from a distance. Whoa a prototype cloaking device ?.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.