Realtime Image Processing for color cameras

There are quite a lot of image sensors with high resolution and high speed capabilities. All of them can generate huge amounts of raw data and it's a tuff problem how to perform image processing and save recorded data in realtime. One can choose simple solution - to use fast SSD drive or RAID to record and save all raw data. In that approach you can quickly ran out of disk space and you will need much time to do image processing to recover high quality images from raw data.

Image sensors of color cameras are covered with Bayer filter arrays to recover color information later at post processing stage. So after demosaicing (debayering) the amount of data will be tripled. Trying to do that even on the fastest CPU we will see that performance for image processing would be far from realtime due to high frame rate or high resolution of image sensors. Below we list some frequently used image sensors to deal with:

CMOSIS image sensors

  • CMV2000: 2048×1088, 340 fps (10 bit), 70 fps (12 bit)
  • CMV4000: 2048×2048, 180 fps (10 bit), 37 fps (12 bit)
  • CMV12000: 4096×3072, 150 fps (10 bit), 90 fps (12 bit)
  • CMV20000: 5120×3840, 30 fps (12 bit)
  • CHR70M: 10000×7096, 3 fps

On-Semi interline transfer CCD image sensors

  • KAI-01150: 1280×720, 138 fps
  • KAI-01050: 1024×1024, 120 fps
  • KAI-02050: 1600×1200, 68 fps
  • KAI-02170: 1920×1080, 60 fps
  • KAI-04070: 2048×2048, 28 fps
  • KAI-11002: 4008×2672, 5 fps
  • KAI-16070: 4864×3232, 8 fps
  • KAI-29050: 6576×4384, 4 fps

On-Semi CMOS image sensors

  • KAC-06040: 2832×2128 (6 MPix), 160 fps
  • KAC-12040: 4000×3000 (12 MPix), 70 fps

On-Semi CMOS high speed image sensors

  • LUPA-300: 640×480, 10 bit, up to 500 fps
  • LUPA-1300-2: 1280×1024, 10 bit, 500 fps

The above image sensors generate raw data in the range of 120 - 960 MPix/s and more. We can offer a solution for realtime image processing and it's possible even for multi-camera systems.

In the preliminary stage we need to define the following:

  • High speed camera parameters: image width and height, fps, operation mode, bit depth (24-bit RGB or 8-bit grayscale)
  • Camera calibration procedures: Dark frame (black offset), FFC, and White Balance (for color cameras)
  • Demosaicing parameters
  • JPEG parameters: compression quality, subsampling mode, restart marker frequency

Data path for realtime image processing on GPU

  • Driver gets raw data from the camera
  • Driver writes data to CPU memory buffer
  • Application sends raw data from CPU to GPU
  • Unpacking module for various camera formats
  • Dark image subtraction
  • Flat-Field Correction
  • White Balance
  • Image demosaicing
  • Adaptive denoising
  • Color correlated temparature and Tint control
  • Image rotation, flip/flop
  • Image crop
  • Image filtering on GPU
  • Image resize
  • Sharp with USM
  • Data transforms with LUT or gamma (10/12/14/16-bit to 8 bits per component)
  • Color correction on GPU with ICC profiles of camera and monitor (option)
  • Image output to monitor via OpenGL
  • JPEG image compression on GPU
  • H.264 compression on GPU (option)
  • Application sends compressed data from GPU to CPU memory
  • M-JPEG and H.264/H.265 integration with FFmpeg
  • Application writes compressed images or video to HDD/SSD/RAID

Contact Form

This form collects your name and email. Check out our Privacy Policy on how we protect and manage your personal data.