GPU Image Processing for XIMEA xiB cameras with PCIe interface

There are quite a lot of image sensors with high resolution and high speed capabilities at the same time. All of them can generate significant amounts of raw data and it's complicated problem how to perform image processing and save recorded data in realtime. One can choose simple solution - just use fast SSD drive or RAID to record and save all raw data. In that approach one could quickly ran out of disk space and you will need much time to do image processing to get high quality images from raw data.

XIMEA ixB series consists of 12-megapixel and 20-megapixel CMOS cameras (color or monochrome) providing an exceptional combination of resolution, speed and image quality.

Image sensors of color cameras are covered with Bayer filter arrays to recover color information later at post processing stage. So after demosaicing (debayering) the amount of data will be tripled. Trying to do that even on the fastest CPU we see that performance for image processing is far from realtime due to high frame rate or high resolution of image sensors. Below we list some frequently used image sensors to deal with:

CMOSIS image sensors

• CMOSIS CMV12000: resolution 4096×3072, 10-12 bits, 130-330 fps
• CMOSIS CMV20000: resolution 5120×3840, 12 bits, 32 fps
• CMOSIS CMV50000: resolution 7920×6004, 8 bits, 30 fps

Data path for realtime image processing on GPU

• Driver gets raw data from the camera
• Driver writes data to CPU memory buffer
• Application sends raw data from CPU to GPU
• Unpacking module for various camera formats
• Dark image subtraction
• Flat-Field Correction
• White Balance
• Image demosaicing
• Base Color Correction
• Denoising
• Curves and Levels with 1D LUT
• Data transform with 3D LUT
• Crop/Rotate/Resize/Remap/Sharp
• Gamma
• OpenGL output to monitor
• JPEG and JPEG2000 compression
• MJPEG/MJ2K stream storage to SSD/HDD/RAID in AVI/MXF

Advantages of GPU image processing for xiB cameras

• Per-pixel offset and shading correction
• High quality demosaicing
• High performance JPEG/JPEG2000 compression
• Low latency output to monitor via OpenGL
• Realtime image processing for the whole pipeline
• Low CPU usage

Benchmarks for GPU image processing

• Hardware: NVIDIA GeForce GTX 1080, CPU Intel Core i7-5930K (Haswell-E, 6 cores, 3.5–3.7 GHz), 16 GB RAM
• Software: Windows-10 (64), CUDA-10.1, Fastvideo SDK 0.15.0.0
• Data for image processing: resolution 4096×3072, 10-bit packed image from CMV12000 image sensor
• Set of parameters #1: DFPD debayer, 8-bit JPEG with quality 90%, subsampling 4:2:0 - time per frame 3.1 ms
• Set of parameters #2: MAD, WB, DFPD debayer, LUT, 8-bit JPEG with quality 90%, subsampling 4:2:0 - time per frame 4.0 ms