It’s plenty easy to do lots of vision tasks with a single camera, but using two cameras gives you lots of magical abilities, like depth perception! If you’d like to experiment with stereo vision, you might enjoy this board that [machinehum] built specifically for hackers.
The project came about because [machinehum] was particularly excited about the Rockchip RV1103, a 1.1 GHz single-core ARM chip. It comes with 64 MB of DDR2 RAM and a has a MIPI CSI-2 bus on board, which comes in handy for interfacing cameras. Thus, it was the perfect part to whip up a stereo vision board with on-board compute capability and Linux support right out of the box. The processor is paired with two OV9281 global-shutter cameras, as well as a LI2HH12 3-axis accelerometer to provide motion sensing. There’s also 4 MB of flash memory thrown on board, as well as a 10/100 Mbit Ethernet interface for good measure. [machinehum]’s write-up also lays out the software stack and how everything works in concert with Video4Linux2.
If you’re new to experimenting with stereo vision, you might find this to be a useful platform for getting started. Design files are on GitHub for the curious. We’ve seen similar projects built using the Raspberry Pi before, too, which itself is essentially just a fast ARM powered computer with useful camera interfaces built-in. Funny that.
I stay on this incredible sight for wonderful things like this… Even tho I stupidly stopped learning and building and making several years ago. So many great ideas and pieces of hardware and software that have come around. I am a bit jealous of you all for staying in such a fun hobby and /or career.
Bravo!
Yes give vision to AI get it out on the street give it a buddy. No faster way for a child to learn. That’s all have a pleasant day I’ve been screaming about vision for AI for a while.
Do you have any measures in place in terms of hardware or software to ensure the cameras run in sync?
I was wondering that as well. I have only read the article, nit delved into the design. If the frames are not synchronised then stereo vision on a moving robot will not be possible.
The RV1103 silicon contains two independent 2-lane MIPI CSI D-PHY receivers allowing each of the OV9281 its own channel, so there is no reason to suspect that they arent simultaneously triggering both global shutters. While the 1.1 GHz single-core Cortex-A7 cannot run high-overhead, pixel-by-pixel stereo processing (like heavy OpenCV block-matching arrays) in standard user space at high frame rates. To make this dual-camera setup functional, you must bypass the main CPU core for vision tasks by leveraging the RV1103’s built-in 0.5 TOPS Neural Processing Unit, a dedicated hardware accelerator designed specifically to execute Deep Learning math (matrix multiplication and convolutions) at ultra-low power.
Instead of forcing the 1.1 GHz single-core ARM CPU to grind through thousands of pixel calculations line-by-line, the NPU acts as a specialized “math co-processor” optimized for AI vision tasks..
You can get depth perception with a single camera as long as there’s any degree of camera movement. You just dont get a scale reference, unless you either see something of known size or have an accelerometer. (acceleration is in m/s^2)
modern slam systems are pretty well developed and typically are developed with the assumption of monocular vision.
Yes. And require 10x compute and RAM to do a similar task. Stereo is hard to beat for object tracking, for ex. It is literally just a bit of geometry. And even if you are doing SLAM it is much much easier to recover from divergences if you have working stereo than just from mono.
Just plain wrong. Have you ever used a SLAM system before?
They date back to the 1980’s. It’s the same math (“just a bit of geometry”) either way, so there’s really no meaningful difference in ram or compute. Hell, binocular vision is significantly WORSE because you now need to perform bundle adjustment on significantly larger matrices describing twice as much input. I don’t know what you mean by divergences, because every slam tracking recovery algorithm works identically in the mono or binocular case.
You also generally can’t assume binocular cameras in most scenarios. Smartphones for example are too physically small for any meaningful binocular baseline.
This appears to be a nice board that will prove useful for a number of stereo vision applications. Two niche application that it does not cover is hyperstereoscopy and hypostereoscopy. With hyperstereoscopy, the distance between the cameras is greater than that for normal human depth perception. One application would be depth perception for autonomous automobiles. With hypostereoscopy, the distance between the cameras is less than that for normal human depth perception. Macrophotography would be the usual application.
A more general hardware solution would allow for placement of the cameras and the processing board on a frame appropriately machined to support the desired camera spacing. A standard frame would support camera spacing for human-like stereo perception. It could be part of the standard design package. A CAD package could be created to enable the production of custom frames that would support camera spacings for the particular hyperstereoscopy or hypostereoscopy application. Ribbon cables would connect the cameras to the processing board.
Is this the processor in the luckfox pico mini?