The Strange Afterlife Of The Xbox Kinect

The tale of the Microsoft Xbox Kinect is one of those sad situations where a great product was used in an application that turned out to be a bit of a flop and was discontinued because of it, despite its usefulness in other areas. This article from the Guardian is a quick read on how this handy depth camera has found other uses in somewhat niche areas, with not a computer game in sight.

It’s rather obvious that a camera that can generate a 3D depth map, in parallel with a 2D reference image, could have many applications beyond gaming, especially in the hands of us hackers. Potential uses include autonomous roving robots, 3D scanning, and complex user interfaces—there are endless possibilities. Artists producing interactive art exhibits would sit firmly in that last category, with the Kinect used in countless installations worldwide.

Apparently, the Kinect also has quite the following in ghost-hunting circles, which as many a dubious TV show would demonstrate, seem almost entirely filmed under IR light conditions. The Kinect’s IR-based structured light system is well-suited for these environments. Since its processing core runs a machine learning application specifically trained to track human figures, it’s no surprise that the device can pick up those invisible, pesky spirits hiding in the noise. Anyway, all of these applications depend on the used-market supply of Kinect devices, over a decade old, that can be found online and in car boot sales, which means one day, the Kinect really will die off, only to be replaced with specialist devices that cost orders of magnitude more to acquire.

In the unlikely event you’ve not encountered non-gaming applications for the Kinect, here’s an old project to scan an entire room to get you started. Just to be perverse, here’s a gaming application that Microsoft didn’t think of, and to round out, the bad news that Microsoft has really has abandoned the product.

Microsoft Discontinues Kinect, Again

The Kinect is a depth-sensing camera peripheral originally designed as a accessory for the Xbox gaming console, and it quickly found its way into hobbyist and research projects. After a second version, Microsoft abandoned the idea of using it as a motion sensor for gaming and it was discontinued. The technology did however end up evolving as a sensor into what eventually became the Azure Kinect DK (spelling out ‘developer kit’ presumably made the name too long.) Sadly, it also has now been discontinued.

The original Kinect was a pretty neat piece of hardware for the price, and a few years ago we noted that the newest version was considerably smaller and more capable. It had a depth sensor with selectable field of view for different applications, a high-resolution RGB video camera that integrated with the depth stream, integrated IMU and microphone array, and it worked to leverage machine learning for better processing and easy integration with Azure. It even provided a simple way to sync multiple units together for unified processing of a scene.

In many ways the Kinect gave us all a glimpse of the future because at the time, a depth-sensing camera with a synchronized video stream was just not a normal thing to get one’s hands on. It was also one of the first consumer hardware items to contain a microphone array, which allowed it to better record voices, localize them, and isolate them from other noise sources in a room. It led to many, many projects and we hope there are still more to come, because Microsoft might not be making them anymore, but they are licensing out the technology to companies who want to build similar devices.

3D Scanning A Room With A Steam Deck And A Kinect

It may not be obvious, but Valve’s Steam Deck is capable of being more than just a games console. Demonstrating this is [Parker Reed]’s experiment in 3D scanning his kitchen with a Kinect and Steam Deck combo, and viewing the resulting mesh on the Steam Deck.

The two pieces of hardware end up needing a lot of adapters and cables.

[Parker] runs the RTAB-Map software package on his Steam Deck, which captures a point cloud and color images while he pans the Kinect around. After that, the Kinect’s job is done and he can convert the data to a mesh textured with the color images. RTAB-Map is typically used in robotic applications, but we’ve seen it power completely self-contained DIY 3D scanners.

While logically straightforward, the process does require some finessing and fiddling to get it up and running. Reliability is a bit iffy thanks to the mess of cables and adapters required to get everything hooked up, but it does work. [Parker] shows off the whole touchy process, but you can skip a little past the five minute mark if you just want to see the scanning in action.

The Steam Deck has actual computer chops beneath its games console presentation, and we’ve seen a Steam Deck appear as a USB printer that saves received print jobs as PDFs, and one has even made an appearance in radio signal direction finding.

Continue reading “3D Scanning A Room With A Steam Deck And A Kinect”

Your Vacuum Cleaner Follows You

There are several projects you can imagine where it would be useful to have a robot follow you. For example, we’ve always wanted luggage that would trail us at the airport and we’ve seen several coolers that will follow you. [Madmax95] apparently dream of having a medical cart following a patient, though, and that’s good too. But how do you do it? [Max’s] method was to strip down a Roomba and build a work table and electronics on it. An Arduino controls the motor and communicates with a PC. The PC reads video from a Kinect camera on the robot and uses special tracking software to follow the patient.

We could easily imagine all of this project except the tracking. That depended on a service called Nuitrack. There is a free version that only works for 3 minutes, but it costs if you want to use it practically. However, it would still be cheaper than rolling your own if your time has value.

Continue reading “Your Vacuum Cleaner Follows You”

This Eyeball Watches You Thanks To Kinect Tracking

Eyeballs are often watching us, but they’re usually embedded in the skull of another human or animal. When they’re staring at you by themselves, they can be altogether more creepy. This Halloween project from [allpartscombined] aims to elicit that exact spooky vibe.

The project relies on a Kinect V2 to do body tracking. It feeds data to a Unity app that figures out how to aim the eyeball at any humans detected in the scene. The app sends angle data to an Arduino over serial, with the microcontroller generating the necessary signals to command servos which move the eyeball.

With tilt and pan servos fitted and the precision tracking from the Kinect data, the eye can be aimed at people  in two dimensions. It’s significantly spookier than simply panning the eye back and forth.

The build was actually created by modifying an earlier project to create an airsoft turret, something we’ve seen a few times around these parts. Fundamentally, the tracking part is the same, just in this case, the eye doesn’t shoot at people… yet! Video after the break.

Continue reading “This Eyeball Watches You Thanks To Kinect Tracking”

Left: kids stomping spiders projected on a driveway. Right: the setup.

Make This Halloween A Spider-Stomping Good Time

We can count on one hand the number of times that we haven’t needed a coat on Halloween night around here. Even if it was fair and sunny the day before, you can count on Halloween being appropriately windy, cold, and spooky. Trick-or-treating only keeps a kid so warm, and we would have loved to happen upon a house with a spider-stomping sugar-burning good time of a game going on in the driveway.

[Kyle Maas] built this game a few years ago, and it has proved quite popular ever since. It’s so popular, in fact, that they have to have someone on duty with a vaudeville hook to yank spectators off the playing field. The point is to stomp as many spiders as you can in a set amount of time, though you only need to stomp one to win. It can handle one to four players, depending on the size of the projection, but [Kyle] says it’s kind of hard to track more than two at a time.

The setup is fairly simple, provided you can reliably affix your projector to something sturdy. [Kyle] used a Structure sensor for the 3D scanner, but you could easily use a Kinect instead. Conversely, the calibration was challenging. [Kyle] ended up using a DSP math trick known as the inverse bilinear transform to be able to calibrate the system using the 3D scanner itself.

If you’re more into scaring the children, just rig up a coffin bell. Either way, don’t forget about our Halloween Hackfest contest, running now through Monday, October 11th. There are more details over on IO. While you’re there, why not check out the list of entries?

Automated Sentry Turret For Your Secret Lab

There are few things as frustrating when you’re trying to get some serious hacking done than intruders repeatedly showing up without permission. [All Parts Combined] has the solution for you, with a Kinect-based robotic sentry turret to keep them at bay.

The system consists of a Microsoft Kinect V2 connected to a PC, which runs an app to do all the processing, and outputs the targeting information to an Arduino over serial. The Arduino controls a simple 2-axis servo mount with an electric airsoft gun zip-tied to it. The trigger switch is replaced with a relay, also connected to the Arduino.

The Kinect V2 comes with SDKs that really simplify tracking human movement, and outputs the data in an easy-to-use format. [All Parts Combined] used the SDK in Unity, which allows him to choose which body parts to track. He added scripts that detect a few basic gestures, issues voice commands, and generates the serial commands for the Arduino. The servo angles are calculated with simple geometry, using XY coordinates of the target received from the SDK, and the known distance between the Kinect and turret. When an intruder enters the Kinect’s field of view it immediately starts aiming at the intruder’s heart, issues a “Hands Up!” command, and tells the intruder to leave. If the intruder doesn’t comply, it starts an audible countdown before firing. [All Parts Combined] also added a secret disarming gesture (double hand pistols), which turns the turret into an apologetic comrade. All it needs is a Portal-inspired enclosure.

It’s a fun project that illustrates how the Kinect can make complex computer vision tasks relatively simple. Unfortunately the V2 is no longer in production, having been replaced by the more expensive, developer focused Azure Kinect. We’ve covered several Kinect-based projects, including a 3D room scanner and a robotic basketball hoop.

Continue reading “Automated Sentry Turret For Your Secret Lab”