Once a popular craze, most of the public has sold or stashed away their plastic video game instruments and forgotten the likes of Guitar Hero and Rockband. Having never been quite satisfied with his scores, [Nick O’Hara] set out to create a robot that could play a Guitar Hero controller. It would be easy enough to use transistors to actuate the buttons or even just a Teensy to emulate a controller and have it play the perfect game, but [Nick] wanted to replicate what it was really like to play. So after burning out a fair number of solenoids (driving them over spec) and learning on his feet, [Nick] slowly began to dial in his robot, Jon Bot Jovi.
The brains of the bot are a Raspberry Pi running some OpenCV-based code that identifies blobs of different colors. The video feed comes from a PS2 via an HDMI capture card. Solenoids are driven via an 8 channel driver board, controlled by the Pi. While it missed a few notes here and there, we loved seeing the strumming solenoid whammy rapidly on the strummer. All in all, it’s a great project, and we love the design of the robot. Whether played by a robot, turned into a synthesizer, or recreated from toy pianos and mechanical keyboards, Guitar Hero controllers offer many hacking opportunities.
Nice project congratulations , How did he capture the video signal ? What hardware did he use for that? Did he use a camera or a video capture card as the article said ? What capture card did he used? And how did he connect it to the raspberry PI?
@ 3 minutes and 10 seconds in the video he show on screen the hdmi to usb video capture device which has been featured in more than one article here on Hackaday… So you can please watch the video for hacks sake? I just came here to write this reply as a rant for your and other comments as a learning opportunity to apply the knowledge I’ve acquired studying with the yt channel “Internet Comments Etiquette” ! Thanks and post!
Hi! I’m Nick, the creator of Jon Bot Jovi. I agree with @danielbpm but I also updated the video description with the links to the capture card and everything else I used in the video. Let me know if you have any other questions!
I get what you were going for, a nice technical challenge. Anyone try to do something similar for a full 4 player band yet? Or for a full band with vocal harmonies included? Now that would be a serious challenge.
Nice job, I like it doesn’t get a 100% score:)
Exactly this was one of the tasks on the “Challenge24” 24-hour programming contest some 10 years ago. The organisers have set up the hardware: solenoids to push the buttons and an IP camera to look at the screen.
As I recall the biggest obstacle to get something working was the timing jitter of the video and control stream, both sent over a shared IP network. Recognizing the notes was actually quite simple, just look for a blob at the appropriate Y height of the video frame. Some filtering was needed to ignore the “flame” effect.
Also using electromechanical relays to control solenoids is asking for degraded precision with such a fast stream of commands.
Taken to extremes, you’d simply use code to reproduce the “guitar’s” data feed to the game after interpreting the video output but at that point the result would be perfect … and boring.
So don’t use a relay but instead use a linear actuator with real time location feedback. Some can have acceleration in the hundreds of meters per second squared. They just are not going to be a few bucks each is all. Or as you indicated, just send the button presses directly.
This current version certainly has some merit but isn’t quite right yet. Needs some better code and perhaps a bit more physical parts as well if the goal is to reach 100% input.
On reflection:
As a perfect-result project this isn’t great, but as an inspiration for readers to actually try things like OpenCV, relays, RPi coding/interfaces etc. without completely breaking the bank, it’s a perfect article.
Future captchas, robots have to make convincingly human accuracy (with errors scattered around accordingly) on guitar hero. I’m going to fail on the low end of the accuracy side and get blocked a lot unless there’s a bass guitar on easy option :)
Guitar Hero ..
Reminds me on South Park :-)
“Real Guitars are for old People”
https://www.youtube.com/watch?v=XX9Q3P0gqpU
I wonder if the bot would’ve done better with double sided usage of the strumming bar, it look like the spring force was not high enough for it to return on the fast notes. Same issue when you play regularly and it’s much easier using both down and upstroke for strumming. Another solenoid and then just a state that shifts using to next one between each note/tick in the song.