Realtime Image Processing for color cameras

There are quite a lot of image sensors with high resolution and high speed capabilities. All of them can generate huge amounts of raw data and it's a tuff problem how to perform image processing and save recorded data in realtime. One can choose simple solution - to use fast SSD drive or RAID to record and save all raw data. In that approach you can quickly ran out of disk space and you will need much time to do image processing to recover high quality images from raw data.

Image sensors of color cameras are covered with Bayer filter arrays to recover color information later at post processing stage. So after demosaicing (debayering) the amount of data will be tripled. Trying to do that even on the fastest CPU we will see that performance for image processing would be far from realtime due to high frame rate or high resolution of image sensors. Below we list some frequently used image sensors to deal with:

CMOSIS image sensors

CMV2000: 2048×1088, 340 fps (10 bit), 70 fps (12 bit)
CMV4000: 2048×2048, 180 fps (10 bit), 37 fps (12 bit)
CMV12000: 4096×3072, 150 fps (10 bit), 90 fps (12 bit)
CMV20000: 5120×3840, 30 fps (12 bit)
CHR70M: 10000×7096, 3 fps

On-Semi interline transfer CCD image sensors

KAI-01150: 1280×720, 138 fps
KAI-01050: 1024×1024, 120 fps
KAI-02050: 1600×1200, 68 fps
KAI-02170: 1920×1080, 60 fps
KAI-04070: 2048×2048, 28 fps
KAI-11002: 4008×2672, 5 fps
KAI-16070: 4864×3232, 8 fps
KAI-29050: 6576×4384, 4 fps

On-Semi CMOS image sensors

KAC-06040: 2832×2128 (6 MPix), 160 fps
KAC-12040: 4000×3000 (12 MPix), 70 fps

On-Semi CMOS high speed image sensors

LUPA-300: 640×480, 10 bit, up to 500 fps
LUPA-1300-2: 1280×1024, 10 bit, 500 fps

The above image sensors generate raw data in the range of 120 - 960 MPix/s and more. We can offer a solution for realtime image processing and it's possible even for multi-camera systems.

In the preliminary stage we need to define the following:

High speed camera parameters: image width and height, fps, operation mode, bit depth (24-bit RGB or 8-bit grayscale)
Camera calibration procedures: Dark frame (black offset), FFC, and White Balance (for color cameras)
Demosaicing parameters
JPEG parameters: compression quality, subsampling mode, restart marker frequency

Data path for realtime image processing on GPU

Driver gets raw data from the camera
Driver writes data to CPU memory buffer
Application sends raw data from CPU to GPU
Unpacking module for various camera formats
Dark image subtraction
Flat-Field Correction
White Balance
Image demosaicing
Adaptive denoising
Color correlated temparature and Tint control
Image rotation, flip/flop
Image crop
Image filtering on GPU
Image resize
Sharp with USM
Data transforms with LUT or gamma (10/12/14/16-bit to 8 bits per component)
Color correction on GPU with ICC profiles of camera and monitor (option)
Image output to monitor via OpenGL
JPEG image compression on GPU
H.264 compression on GPU (option)
Application sends compressed data from GPU to CPU memory
M-JPEG and H.264/H.265 integration with FFmpeg
Application writes compressed images or video to HDD/SSD/RAID