Ask Hackaday: What Are We Going To Do With The New Kinect?

kinect

Yesterday Microsoft announced their new cable box, the Xbox One. Included in the announcement is a vastly improved Kinect sensor. It won’t be available until next Christmas, but now the question is what are we going to do with it?

From what initial specs that can be found, the new version of the Kinect will output RGB 1080p video over a USB 3.0 connection to the new Xbox. The IR depth camera of the original Kinect has been replaced with a time of flight camera – a camera that is able to send out a pulse of light and time how long it takes for photons to be reflected back to the camera. While there have been some inroads into making low-cost ToF cameras – namely Intel and Creative’s Interactive Gesture Camera Development Kit and the $250 DepthSense 325 from SoftKinetic – the Kinect 2.0 will be the first time of flight camera you’ll be able to buy for a few hundred bucks at any Walmart.

We’ve seen a ton of awesome Kinect hacks over the years. Everything from a ‘holographic display’ that turns any TV into a 3D display, computer vision for robots, and a 3D scanner among others. A new Kinect sensor with better 3D resolution can only improve existing projects and the time of flight sensor – like the one found in Google’s driverless car – opens up the door for a whole bunch of new projects.

So, readers of Hackaday, assuming someone can write a driver in a few days like the Kinect 1.0, what are we going to do with it?

While we’re at it, keep in mind we made a call for Wii U controller hacks. If somebody can crack that nut, it’ll be an awesome remote for robots and FPV airplanes and drones.

60 thoughts on “Ask Hackaday: What Are We Going To Do With The New Kinect?

  1. Here’s my question: Will Kinect 2.0 be available for individual sale? I got the impression that it would be bundled with the Xbox One from the unveiling presentation. It may be available separately as a replacement for broken units due to end user error ie. dropping, crushing, etc. I am curious as to how this will be handled. I can only hope they make it readily available. Their initial efforts with the “Kinect 1.0” were disappointing in the beginning.

    -Alex

    P.S. I also suspect it won’t be cheap to replace.

  2. i cant wait! … $250 is used to be what i payed for a 1080P webcam alone … now with all of this? even fucking TOF?! i cant wait to hook one of these up to my PC!

    1. tv, tv, tv, television, tv, tv, tv, sports, sports, sports, sports, tv, sports, tv, tv, television, tv, call of duty, call of duty, dog, dogs, dog, call of duty, call of duty, tv, tv.

      1. read, read, read, book, read, bike, bike, bike, bike, coveralls, read, read, manual, read, read, blueprints, work, work, work, work, shorts, shirt, bike, bike, bike, bike, groceries, bike, bike, bike, bike, garage, coveralls, hobby, hobby, hack, hack, ratchet, hammer, read, read, read, hackaday, read, read, read, instructables, read, read, read, hammer, hammer, grind, grind, weld, weld, weld, weld, hammer, hammer, scrape, grind, wires, solder, solder, cool down, living room, water, splash, splash, feed fish, read, read, read, read…

  3. Judging by how much DRM Microsoft is putting on everything else Xbox 3.0 I would imagine this will be a tough hardware crack. Also USB 3.0 requires much faster and more expensive sniffing hardware so that limits the number of people even capable of doing it.

    1. Well, if they announced it will be available for the PC too then it makes sense it will come with a SDK so that should deal with most of those issues I would say.

      Then there is still the pricing though..

    2. Except that your missing that Microsoft has from the beginning encouraged highschool/college students to play with and find new ways to use the original Kinect. There’s no reason to think they would be any different with the second Kinect.

  4. Well, can’t be worse than last time.

    “Here’s the PC SDK we promised after you backed us into a corner…except it only /really/ works with the new PC Kinect we just started selling for twice the price, even though we know you all bought the Xbox 360 one to hack already!”

    What the hell was that even about? Then again, I guess I shouldn’t be surprised, the Xbox division’s idea of ‘customer service’ has historically been “Turn ’em upside down, shake ’em ’til money stops coming out, and then screw ’em with their pants on!”

      1. NOW it does.
        When they first released the sdk, it did not recognize the 360 version and the pc version was $250 instead of $100. Microsoft was trying to market the pc version to a different market and the 360 version was subsidized by game sales, so they didn’t want hackers and developers to buy the subsidized version instead. The open source drivers made this irrelevant, as everyone just bought the cheaper version and used the unofficial driver instead of shelling out the extra $150. It took less than a month for MS to drop the PC version’s price and update the sdk to use either unit.

  5. I am still looking at Kinect and wonder what is it really it does a ton of awesome things and none of them really related to gaming. It’s suppose to be about gaming right? yet seems to be better at everything else except games.

    1. I agree. Hold your hands in front of you, the Kinect finds your fingers, and you’ve got everything from a keyboard to a game pad and the swiping motions of the smartphone all in one. When playing a magical game, you’d be actually waving your hands and chanting an incantation when casting a spell. Point your finger and say bang.

    2. Why not a Leap. It’s smaller and does the same thing in a shorter, more useful range (compared to the console kinect, because the PC version is tuned to work closer)

      1. Well, I didn’t actually ment kinect for controls, I ment kinect for surrounding awarenes. My idea was for kinect to scan the room you are in, and turn it, say into castle walls, where you can walk around it.

  6. I guess it depends on a few things:

    1) What’s the range of depth tracking? Not sure about the limitations of ToF in general, but the DepthSense 325’s max range for example is pitiful compared to the original Kinect.
    2) Can it be used with USB 2.0 at reduced resolution if needed, or are you stuck with connecting it only to high-power computers and equivalents?
    3) How nice is Microsoft going to play this time around?

    1. All good points.

      On a plus side we get real 720p pixels of depth map(maybe even at 60Hz?), on the negative we get less range and probably drm up the ass.

      Seems Kinect 2 will finally be the result of M$ spending up to a BILLION (:o) dollars buying up companies before X360 launch. They went on a spending spree (3DV, Canesta) and a year later realized they still couldnt make it work – thats why M$ licensed Kinect from PrimeSense.

      1. True that. But let’s say you don’t need full 1080p video or skeletal tracking. You only want to experiment with low-res ToF depth sensing on a Raspberry Pi, for example. And don’t want to buy a separate (and more expensive) sensor to do it. Supporting reduced functionality over USB 2.0 would be a mighty nice feature.

  7. Sentry turret. A high resolution range finding people identifier would make for a heckuva sentry turret. Add that fancy force prediction stuff and

    The high resolution of the ToF camera would make a nice 3d scanner.

    ROBOTS! ToF cameras make a great deal of robotics problems trivial. Make an autonomously navigating turtle with a robot arm on it to fetch you drinks.

    Make a robot with active suspension that uses the ToF camera to properly conform to the terrain ahead. It’d be just like a ghetto version of Neal Stephenson’s smart wheels.

    1. Yes a turtle! It’s world turtle day.
      That Native American saying about the earth rests on the back of a giant turtle, looks right as seen from orbit or transit beyond.

  8. If this is really a ToF camera, it means that there will finally not be the “shadow” on the border of objects that was so prevalent under the kinect 1. I am very skeptical however over the possibility that ToF will improve much more than that the performances of the kinect. Wait and see…

      1. Given there’s dozens of youtube videos and a SDK, what makes you think its vapourwape?
        Thats one hell of a mockup too, if all the figure tracking pointcloads are faked.

    1. No clue what the SDK gives you, but it clearly works with a pointcload internally as there’s demos of that with two active hands moving about in realtime.

      1. At least some of that point cloud is faked. It obviously can’t see the top of the hands, and generates the points from them by interpolating ellipses from the bottom. Giving an effect a bit like an articulated balloon animal. Or maybe a hand made of sausages. But that much is expected.

        Curiously though, some features seem to be missing from the bottom of the hand too. Everything, top and bottom, is sliced into perfect ellipses if you look very close. Which can’t represent – and in fact don’t show – the negative curvature of the cup of the palm. I have no idea how that sensor works or what it’s actually returning, but the point cloud they show ain’t it.

  9. btw, the Wii U gamepad has had some promising work:
    http://www.eurogamer.net/articles/digitalfoundry-secrets-of-the-wii-u-gamepad
    (from TFA)
    “We modified wpa_supplicant/hostapd to have it work with the non-standard things, and were able to pair a PC with a Wii U that way. This took us less than a week – our time since then has been spent reverse engineering the custom communication protocol used between the Wii U and the GamePad.”

    1. But then better things will be widely available and it’s too old to be interesting.

      (As always that is assuming that western society remains somewhat stable and free over the longer period.)

  10. The new camera looks very interesting. We built the Holodome which was mentioned earlier in the comments, and have recently joined forced with another company to work on a new Volumetric product called Voxiebox. We’ve been using experimenting with various depth cameras and also LEAP in order to capture and display real time volumetric data and there are pros and cons with each. The LEAP tracks finger movement, and is great for interacting with a display. Its not a depth camera.

    The kinect is great for large room data capture. It does suffer from shadowing caused by using two sensors to capture the data, and also has poor spacial resolution up close.

    The Intel camera is wonderful, very solid and uses time-of-flight which gives it a true 320*240 resolution with no shadows. This makes it ideal for close up , very detailed point capture.

    The new new kinect appears to be similar and although the RGB feed from the cameras is 1080p, the time of flight data is likely to be much less, possibly 640*480, but even that will be fantastic for detailed depth capture.

    Heres a short video of our latest prototype volumetric display. Its show playing games, and also rendering interactive chess using Leap Motion.

    If the link is suppressed, just search youtube for ‘Voxiebox’

    http://www.youtube.com/watch?v=1ydSXboRjQo

    1. There is a video about the new kinnect here http://www.theverge.com/2013/5/21/4353232/kinect-xbox-one-hands-on/in/4116279

      And in that video you see the IR lighted room for a second and also what appears to be the resulted ToF data captured later on..

      And from a comparison its clear the IR light source is the one used to enable the ToF, which as you say isn’t full HD (no surprise). And interestingly you see the light falloff in the corners on the plain IR shot and the same increased error in the corners of the captured ToF data, and you can also see some materials create more confusion due to adsorption of the IR light I expect. Makes you wonder what plantlike would look like to it, and spandex…
      Another thing is that from those shot it shows doesn’t do the laser-dot-pattern but a smooth IR light illumination because of the changed way of capturing data, maybe it’s just regular IR LED now?

      Incidentally that article says the IR light kicks in when it’s too dark, but seeing ToF needs a controlled known lightsource AFAIK I expect the author got that wrong and the IR is modulated and always on.

      Now once it’s there for hacking people can find out if you can mess with that lightsoucrrce and increase its range and add a zoomlens to the camera to do remote kinnecting, but those are some very preliminary wild HaD thoughts :)

  11. I am suprised no one has commented yet on the killing of used games. even the answers microsoft has given is worring. They want to take over the used market pretty much.

  12. Nobody’s mentioned that since this is ToF you could hack in a low-res backscatter scanner? Wow.
    Some of the demo videos show that the torso of people looks grainy. This is because some of the photons are passing through the shirt and bouncing off the underlying muscle. Apply a few statistical methods and voila – privacy-shattering technology at your fingertips!
    Granted, there’s no way it would be NEARLY as detailed as the TSA scanners. Those use microwaves which pass through clothing much more readily.

Leave a Reply to WhatnotCancel reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.