Going to the park with your augmented reality girlfriend

Lonely? Bored? Really into J-pop? If you’re any of these things, here’s the build for you. It’s an augmented reality system that allows you to go on a date with one of Japan’s most popular virtual singers.

The character chosen to show off this augmented reality girlfriend tech is [Hatsune Miku], a voice synthesizer personified as a doll-eyed anime  avatar. [Miku] is an immensely popular character in Japan, with thousands of people going to her concerts, so choosing her for this augmented reality girlfriend project was an obvious choice.

The build details for this hack are a little sparse, confounded by the horrible Google Translate results of the blog linked in the YouTube description. From what we can gather from the video and this twitter account, the build is based on an ASUS Xtion Kinect clone and a nice pair of video goggles.

We’re expecting the comments for this post to fill up with, ‘Japan is really weird’ comments, but we can see a few very, very cool applications of this tech. For instance, think how cool it would be to be guided around a science museum by [Einstein], or around Philadelphia by [Ben Franklin].


  1. Mr. Hsu says:

    This reminds me way too much of Dr. Krieger (from Archer) and his virtual Japanese bride.

  2. Mike bradley says:

    Yep, HAD shows most nerds never talk to girls.

  3. Bob D says:

    He we full Hacker. Can’t find an appropriate girlfriend? Make one!

    Very cool project. My first thought was also virtual docent tours, but I could see it in more formal educational environments too. (I’d trade my crappy class-assigned lab partner for Darwin, Hooke, or Galileo in an instant.)

  4. Gdogg says:

    Japanese people are weird as fuck.

  5. Bryan W says:

    Combine this with speech recognition and network connectivity and you’ve got yourself the next stage in home automation.

  6. Redion says:

    Virtual Porn here we come LOL!!!!!!! :-)

  7. escott says:

    Or, why not cruise around The Franklin Institute (in Philly) guided by… Frankling and Einstein. You know, just a thought.

  8. matt says:

    That guy is awesome, he actually programed it so when you hit her on her head she flinches, just like a real woman. How long until someone combines this with the video game Rapelay?

  9. la89ondevg says:

    I guess this could be helpful for socially isolated people (stressed/traumatised) or for counselling of patients/prisoners who could hurt a real person. Or more specifically having a virtual presence of loved ones who aren’t anymore.

  10. la89ondevg says:

    This could be really awesome hack. It can be used for healing of traumatised/stressed patients or prisoners which can hurt themselves or their counsellor. And most interestingly it can be used making a virtual presence of loved ones which aren’t available or not anymore.

    • Katie says:

      or it could be combined with internet and linked to a canmera that looks at the counselor, and wallah! You’ve got the nearest thing to Star Wars-style hologram communication. The counselor could communicate with the patient but be totally safe. Or this tech could be a cool way of talking to far awat friends. Forget video chat, you could see your friend right in your world aqnd interact with her.

  11. Dra says:

    Or went on an virtual date with your actual girlfriend in different parts in different parts of the world. But holy FUCK japan is creepy.

    • Bill Stewart says:

      Yeah – from the article title, I was expecting something like “My girlfriend made these really cool Augmented Reality Glasses (or worked on Google Glass), and we went to the park where she told me the names of all the people and plants we saw.”

  12. filespace says:

    The best use of this is porn..end of story..lol live porn on your Living room floor…via some Goole google type device

  13. Reboot says:

    The main problem I think with this form of augmented reality is stability. Adding more tracking features would probably help.

    There’s augmented reality in a game (with Hatsune Miku) on the 3DS, using some sort of cards:

  14. Ryoku says:

    While this application is unsettelingly strange the concept at heart is interesting and shows a lot of promise for future gaming in the real world just so long as reality is discernible from the augmented reality counterparts. The simulation is a bit buggy in the video, we in the industry/hobby of CG have taken great pains to keep characters from sliding and popping in 3D and there are piles of noticeable artifacts from this process but they seem to be more a matter of hardware than coding or animation (with some exception) then there is the matter of range and sensitivity that come into play with reality clipping (such as the scene where the tree occludes the AR character or the demonstrators arm) one would almost need two maybe three depth cameras synced to the rgb capture. I think it’ll be a few years before we can start going to our local park to quest WOW or have a virtual Easter egg hunt in our back yard jungle (assuming you live on a dirt lot like I do and the jungle in question would be AR CG)

    Snow Crash anyone?

  15. Tron9000 says:

    ok…..so Hack-a-day missed out the single most important discovery in the history of physics at cern and yet reports on some socially inept, anime fans over engineered wet dream!!!!???

  16. macintux says:

    I love J-pop just as much as the next guy, but this is just fucking sad.

  17. Derrick says:

    Ignoring the actual content here, this is awesome from a hacker perspective and has fantastic implications. Telepresence, etc, there’s enormous potential.

    The actual implementation?

    Obligatory: Japan, WTF?

    Also: Seriously, dude. You were sure to code her responding to being groped and hit in the head. This is why you have to make your own girlfriend.

  18. Daniil says:

    Soon we’ll be seeing people running around pretending to hold guns and swords, playing their favorite fps/mmorpg in real settings. I say bring that shit on.

  19. mac says:


    my japanese isnt 100% but from looking at the video, and applied some logic

    he used several off the shelf components, pc, kinect like hw, some oss motion and dimension tracker(ptam) and a lot of coding to make this happened.

    all done by a single guy

    imagine what kind of potential it could reach when you have such a system developed by well finance and research company. it would well exceed google’s project glass on so many different level.

    imagine what kind of application you could use it for,
    education, simulation, medicine, engineering, navigation, natural interface, entertainment, human interaction.

    i assumed the use of ir depth sensor (kinect), is to detect the tree and his hand that are “infront” of the virtual model. since ptam can only tell the dimention space in reference to where the viewer is to.

    if you combine it with better depth sensor, gyroscope, accelerometer, barometer, compass, tiny computer, battery, gps, microphone, speaker, etc. you would build killer application.

    if it can be made it into one small package like google glass at a affordable price, it would change how we would interact the world, the concept of current computing interface would just feel old and inadequate.

    our imagination, concept and expression would no longer confined to a 2d plane (or fake 3d on a 2d screen), but full 3d space. you can bring up a 3d model infront of you, analyze data in a way that you cannot or very hard to visualize. like looking at protein structure, virus, black hole, planet orbit, etc.

    this can all actually be achieved by current technology, all you need to do is mass produce to make it cheaper and have everyone using it. whats left are open standards such as interface, api, models, protocols, etc. something like iso. if not, we would be stuck with millions of different system which cant be interoperated. imagine how suck it would be that you want to load a data set for your cnc model, but realized that it is locked behind some drm, or vendor specific hw/sw (till you switch or pay).

    i see this is one small step in really awesome direction, and will keep a close watch in the ar/vr development.

    • JB says:

      As long as they don’t ever make Navi to annoy you at every turn :P

    • Craig Turner says:

      Augmented reality is a very exciting field. Especially in getting people to play again in a physical world eg outside. Imagine what this tech could bring to virtual hunting, paint-balling and air-soft games etc… Education eg Augmented reality zoos with extinct animals in augmented environments. I hope a big development house picks this up.

  20. Threeck says:

    Use it long enough and you’ll be seeing Hatsune where ever you go. with or without the glasses.

  21. ant says:

    At last teachers are obsolete! and now my “friends” are exactly how i want them to be.. minus the asshole element they all have now! thanks japan!

  22. nikescar1 says:

    I really, really love this tech and the implementation, but that doesn’t make this sort of thing any less creepy.

  23. anglophony says:

    Sahdes of W.Gigson’s Idoru.

  24. Joe Bonasses says:

    Closest culture to aliens on the this planet. That’s all I have to say about the Japanese….

  25. Necromant says:

    I guess I’m oldscool, but I’d prefer a real girlfriend

  26. Mikey says:

    Or, how cool it would be to go on a date with your real girlfriend while she’s out of the state getting her masters degree at Cornell.

    * thinks about that statement for a while *

    I mean… with you each having a set of the hardware of course and the date being coordinated in real time and over the internet and all that. Not like, going on a date with a virtual version of her, that’s just weird.

  27. Roger Wilco says:


  28. Jarina Roy says:

    Hi everyone! Imagine what kind of application you could use it for,education, simulation, medicine, engineering, navigation, natural interface, entertainment, human interaction.

  29. Mythic says:

    The possible applications of this technology are amazing, many of which the comments have already pointed out. The Miku girlfriend thing is probably just a way to help catch the attention of an internet audience and Japan’s popular culture.

    But that aside, this is just another sign that 2013-2015 will be the dawning years of augmented reality, and though we still have obviously large leaps and bounds to make.

  30. Al says:

    It might help people to know that Miku is a standard reference character used by those working in AR/VR and 3D display technology in Japan. And with good reason. Aside from her immense popularity, her name and image are freely licensed for noncommercial use; she has her own open source 3D rendering software MikuMikuDance which has become the most widely used 3D animation program in Japan (search for [MMD] on youtube), and there are lots of freely available 3D models of Miku.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

Join 96,441 other followers