Getting 1000 FPS Out Of The Raspberry Pi Camera

The Raspberry Pi camera has become a de facto standard for many maker projects, making things like object recognition and remote streaming a breeze. However, the Sony IMX219 camera module used is capable of much more, and [Gaurav Singh] set out to unlock its capabilities.

After investigating the IMX219 datasheet, it became clear that it could work at higher bandwidths when configured to use all four of its MIPI CSI lanes. In the Raspberry Pi module, only two MIPI lanes are used, limiting the camera’s framerate. Instead, [Gaurav] developed a custom IMX219 breakout module allowing the camera to be connected to an FPGA using all four lanes for greater throughput.

With this in place, it became possible to use the camera at framerates up to 1,000 fps. This was achieved by wiring the IMX219 direct to an FPGA and then to a USB 3.0 interface to a host computer, rather than using the original Raspberry Pi interface. While 1,000 fps is only available at a low resolution of 640 x 80, it’s also possible to shoot at 60 fps at 1080p, and even 15 fps at 3280 x 2464.

While it’s probably outside the realm of performance required for the average user, [Gaurav] ably demonstrates that there’s often capability left on the table if you really need it. Resources are available on Github for those eager to delve deeper. We’ve seen others use advanced techniques to up the frame rate of the IMX219, too. Video after the break.

 

34 thoughts on “Getting 1000 FPS Out Of The Raspberry Pi Camera

  1. (Grabs dslr camera, scrolls through settings) 30s, 20s, 10s, 5s, 3s, 1s, 5fps, 10fps,15fps, 30fps, 60, 100, 500, 1000, 2000, 3000, 4000, 5000. Neat project though perhaps I could do something similar with my one of my older cameras.

  2. You can capture 640×75@1007fps WITHOUT any additional hardware, just with v2 camera connected to CSI-2 interface of Raspberry Pi, and many other framerates for bigger frames, eg. 640×224@410fps:
    https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=212518&p=1310445#p1320034

    You can find many examples captured in my Raspberry camera gallery:
    https://github.com/Hermann-SW/Raspberry_camera_gallery#raspberry-camera-gallery

    For people thinking 640×75 is not useful, this was first 1007fps video I took of closing classical mousetrap bar. Closing takes 1/100th second in total, but with capturing 640×75@1007fps you will see 11 frames, first being last unreleased mouse trap bar, last being closed (played at 1fps):
    https://stamm-wilbrandt.de/en/forum/mt.1000fps.75.gif

      1. That is correct.
        With 30s.
        That is more than enough for eg. capturing closing mouse traps or other fast moving stuff.
        I agree that your solution is good to have when needing to capture minutes or hours @1000fps.

        What is the price one has to pay to build your solution?

        Can your solution be adapted to capture v1 camera ov5647 MIPI traffic as well?
        You can only get 640×64@665fps from that.
        But you can make v1 camera do global external shutter capturing (just by setting some registers):
        https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter#introduction

        That would avoid the extreme rolling shutter effect that v2 camera always has, eg in this video (the closing bar should be vertical):
        https://raw.githubusercontent.com/Hermann-SW/Raspberry_camera_gallery/master/res/mt.1007fps.10000lm.12us.anim.gif

        1. Solution Suggested here is also relatively cheap when compared to it’s FPGA implementation couterpart. Lattice Machxo3 is really quite cheap generic FPGA. When Compare to what FPGA vendors offer to be used with for Image solution it is order of magnitude less.
          While looking at Work by Hermann, it is exceptional to be able achieve such a performance on the PI itself. But limitation Raspberry PI itself limits the applications as well. Comparing this solution to a Raspberry PI would not be in the correct direction.
          It is a totally different class of solution.
          It open up the door of using many other High performance Image sensor to be used in Open source projects.

        1. Hi, if you can tell me it would be very appreciated.
          No matter what, the maximal framerate was always 1007fps.
          It seems that I miss some register setting in that case.
          I determined this formula for v2 framerate: 76034/height+67
          This is quite goot for height=96: 76034/96+67=859, while real framerate is 841fps.
          I really would like to capture 640×64@1255fps according that formula (or better 640x128_s@1255fps with capturing only every other line and then doubling each line in post processing):
          https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=212518&p=1310445#p1320034

          One application would make the FPGA solution presented here not needed even for hour long capturing is real time frame processing on a fast moving robot. That way the frames do not need to get stored. I never tried, but sending frames via Pi3B+/Pi4B fast WLAN to other computer similar to this FPGA approach would allow for the same arbitray long capturing of 640×75@1007fps:
          $ echo “640*75*10/8*1007/1024^2” | bc -q
          57
          $
          Pi4B Wifi can do 58Mbps@2.4GHz and 114Mbps@5GHz Wifi.

          1. I did saw Frame rate upto ~1500 FPS with Maximum MIPI output freqency all max possible ADC pll at around 640×80, right now because of pipeline and memoryless output reformatter fpga module This FPGA solution limited with because of GPIF frequency of FX3. If either FX3 could have gone higher than 32bit 100Mhz , or i would implement FIFO in output reformatter it can certainly got upto at least 1500FPS

    1. This Project is targeted at a totally different class of solution.
      Solution with FPGA is different from Raspberry PI. Goal of the project was to be able to use a MIPI CSI-2 camera at first with FPGA and then later implement intelligent solution on FPGA itself to do something useful with it
      As other have pointed out it has some application in robotics vision at high speed.

  3. 640 x 80? You wanted to say: 640 x 480
    And why on earth would you want 1000 FPS when the eye cannot tell the difference anymore over 30. It’s a nice show off in video game that look at man crysis is running with 100FPS whoa where it would look the same if it would be running with 50 so WHY?

    1. What the hell are you talking about? Eye cant see past 24 fps, thats why Cinema and PAL both use ~this frequency. You got brainwashed by National Television System Committee and their wasteful 30 frame refresh rate conspiracy. Everyone knows those extra 6 frames are allocated for subliminal transmissions.

  4. Most excellent article! Do you think it would be possible to use 2 (or more) cameras on the same FPGA?
    If some way the FPGA could trigger them in sequence you could possibly double the (virtual) framerate

      1. USB FX3 can do 32bit 100Mhz, which translate to 400 MBytes Per Second, a little over clocking of FX3 GPIF upto around 112MHz will bring 448MBytes Per second. This bandwidth in theory Translates to at 640×80 to ~4000 FPS. FX3 is quite nice Chip i have found it delivers most of the time more than it claims. USB controller on FX3 can do full what USB 3.0 allows but 32bit Parallel port (GPIF) limits it to as mentioned above to 32bit @ 100MHz.
        Cypress offers Other FX3 like controller CX3, CX3 is specifically for MIPI CSI camera. CX3 can active full USB 3.0 bandwidth. But one must implement custom application on PC side to receive RAW video data as CX3 does not have any logic to do any Debayer or for UVC, RGB to YUV conversion. (UVC neither support RAW nor RGB)

        Idea with Double camera may be possible, Built in PLL on camera may cause issue and drift frames, but in theory FPGA can keep track of Frames and correct if necessary. A somewhat larger FPGA would be needed, Lattice MachXO3 is a relatively small FPGA if looked for crucial high speed resources. Multi camera design will LUT/Gate wise fit onto MachXO3 but because of limited clock resources will cause issues doing anything fast.

        Multi USB Design is certainly possible but need lot of custom solution on both of Hardware and PC side. Generally when designer surpass what USB can offer, they go to PCIE

        Lowest hanging fruit for optimization is output refromatter module in FPGA Right now FPGA YUV output is running at full 100Mhz ( which is driven by 200Mhz mipi clock /2 ) but output is fragmented and does not take full advantage of 100Mhz FX3 bus. If i would make a FIFO in reformatter we can de-couple MIPI clock and FPGA output. which make possible to run MIPI clock faster than 200Mhz. Wich ultimately makes it possible to get even faster frame rate, i have seen frame rate @ 640×80 to upto 1500 FPS when i ran MIPI clock faster form 270 to 320Mhz. Hardware in the current state may hinder reaching high MIPI frequency. May need to have custom FPGA board with correctly terminated CSI lanes.

        Second optimization would be again in same reformatter but little complicated. buffer a lot more data and utilze 100 % of 32bit FX3 Bus. In theory one can utlize some other USB controller and get all bandwidth what USB 3.0 allow.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.