Camera Development

As open as the Raspberry Pi Foundation has been about their beloved products, they would be the first to admit there’s always more work to be done: Getting a Pi up and running still requires many closed proprietary components. But the foundation works to chip away at it bit by bit, and one of the latest steps is the release of a camera stack built on libcamera.

Most Linux applications interact with the camera via V4L2 or a similar API. These established interfaces were designed back when camera control was limited and consisted of a few simple hardware settings. Today we have far more sophisticated computational techniques for digital photography and video. Algorithms have outgrown dedicated hardware, transforming into software modules that take advantage of CPU and/or GPU processing. In practice, this trend meant bigger and bigger opaque monolithic pieces of proprietary code. Every one a mix of “secret sauce” algorithms commingling with common overhead code wastefully duplicated for each new blob.

We expect camera makers will continue to devise proprietary specialties as they seek a competitive advantage. Fortunately, some of them see benefit in an open-source framework to help break up those monoliths into more manageable pieces, letting them focus on just their own specialized parts. Leveraging something like libcamera for the remainder can reduce their software development workload, leading to faster time to market, lower support cost, and associated benefits to the bottom line that motivates adoption by corporations.

But like every new interface design borne of a grandiose vision, there’s a chicken-and-egg problem. Application developers won’t consume it if there’s no hardware, and hardware manufacturers won’t implement it if no applications use it. For the consumer side, libcamera has modules to interop with V4L2 and other popular interfaces. For the hardware side, it would be useful to have a company with wide reach who believes it is useful to open what they can and isolate the pieces they can’t. This is where the Raspberry Pi foundation found a fit.

The initial release doesn’t support their new High-Quality Camera Module though that is promised soon. In the short term, there is still a lot of work to be done, but we are excited about the long term possibilities. If libcamera can indeed lower the barrier to entry, it would encourage innovation and expanding the set of cameras beyond the officially supported list. We certainly have no shortage of offbeat camera sensor ideas around here, from a 1-kilopixel camera sensor to a decapped DRAM chip.

[via Hackster.io]

The hottest new trend in photography is manipulating Depth of Field, or DOF. It’s how you get those wonderful portraits with the subject in focus and the background ever so artfully blurred out. In years past, it was achieved with intelligent use of lenses and settings on an SLR film camera, but now, it’s all in the software.

The franken-camera rig, consisting of five Pixel 3 smartphones. The cameras are synchronised over WiFi.

For the Pixel 2 smartphone, Google had used some tricky phase-detection autofocus (PDAF) tricks to compute depth data in images, and used this to decide which parts of images to blur. Distant areas would be blurred more, while the subject in the foreground would be left sharp.

This was good, but for the Pixel 3, further development was in order. A 3D-printed phone case was developed to hold five phones in one giant brick. The idea was to take five photos of the same scene at the same time, from slightly different perspectives. This was then used to generate depth data which was fed into a neural network. This neural network was trained on how the individual photos relate to the real-world depth of the scene.

With a trained neural network, this could then be used to generate more realistic depth data from photos taken with a single camera. Now, machine learning is being used to help your phone decide which parts of an image to blur to make your beautiful subjects pop out from the background.

Comparison images show significant improvement of the “learned” depth data versus just the stereo-PDAF generated depth data. It’s yet another shot fired in the smartphone camera arms race, that shows no signs of abating. We just wonder when the Geiger counter mods are going to ship from factory.

[via AndroidPolice]