GPU Software for Machine Vision Cameras

Author: Fyodor Serzhenko

Cameras for machine vision, industrial and scientific applications are quite widespread nowadays. These solutions utilize USB3, GigE, 10GigE, 25GigE, Coax, Thunderbolt, PCI-Express interfaces to send data from camera to PC. Usually cameras transfer RAW data and we need to implement quite complicated image processing pipeline to convert RAW to RGB in realtime. This is computationally heavy task, especially for high speed and high data rate cameras.

gpu software machine vision

Realtime image processing for machine vision cameras could be done on Intel/AMD CPUs, but that solution is difficult to speedup further, which is the case for multicamera systems. To overcome such a bottleneck, one could implement simplified algorithms on CPU to be on time, but it can't be a good solution. Most of high quality algorithms are slow even at multicore CPUs. The slowest algorithms for CPU are demosaicing, denoising, color grading, undistortion, resize, rotation to an arbitrary angle, compression, etc.

To solve that task we utilize Fastvideo SDK which is working on NVIDIA GPU. With that SDK we can implement a software solution where full image processing pipeline is done on graphics processing unit (GPU). In that case the software architecture is much more simple, because processing part is fully done on GPU and it doesn't interfere any more with CPU-based features.

Fast CinemaDNG Processor software

As a good example of high performance and high quality raw image processing we could recommend Fast CinemaDNG Processor software for Windows, where all computations are done on NVIDIA GPU and its core is based on Fastvideo SDK engine. With that software we can get high quality image processing according to digital cinema workflow. One should note that Fast CinemaDNG Processor is offering image quality which is comparable to the results of raw processing at Raw Therapee, Adobe Camera Raw and Lightroom Photo Editor software, but significantly faster. Total speedup could be estimated as 10-20 times or even more.

To check GPU-based performance for your machine vision camera on Fast CinemaDNG Processor software, you can transform your RAW images to DNG format with our open source PGM2DNG converter, which could be downloaded from Github. After such a convertion the software will be able to work both with digital cinema cameras like Blackmagic Design and with machine vision / industrial cameras.

machine vision camera sdk gpu

GPU Camera Sample Project

Here we will show how to implement a software with GPU image processing pipeline for any machine vision camera. To accomplish the task we need the following:

  • Camera SDK (XIMEA, JAI, Imperx, Flir, etc.) for Windows
  • Fastvideo SDK (demo) ver.0.15.0.0 for Windows
  • NVIDIA CUDA-10.1 for Windows and the latest NVIDIA driver
  • Qt ver.5.13.1 for Windows
  • Compiler MSVC 2017

Source codes and links to supplementary libraries for gpu-camera-sample project you can find on Github.

As a starting point we create a software to capture raw images from XIMEA cameras. We can easily find such a sample application at XIMEA SDK and we can incorporate that code into our software. In the software we utilize default camera parameters to focus on GPU-based image processing, though you can add any GUI to control camera parameters as well.

Simple image processing pipeline on GPU for machine vision applications

  • Raw image capture (8-bit, 12-bit packed/unpacked, 16-bit)
  • Import to GPU
  • Optional raw data convertion and unpacking
  • Linearization curve
  • Bad pixel removal
  • Dark frame subtraction
  • Flat-field correction
  • White Balance
  • Exposure correction (brightness control)
  • Debayer with HQLI (5×5 window), DFPD (11×11), MG (23×23) algorithms
  • Wavelet-based denoising
  • Gamma
  • JPEG compression
  • Output to monitor with minimum latency
  • Export from GPU to CPU memory
  • Storage of compressed data to SSD

It's possible to modify that image processing pipeline according to your needs, as soon as source codes are available. There are much more image processing options in the Fastvideo SDK to implement at such a software for GPU-based image processing in camera applications.

The software has the following architecture

  • Thread for GUI and visualization (app main thread)
  • Thread for image acquisition from a camera
  • Thread to control CUDA-based image processing
  • Thread for OpenGL rendering
  • Thread for async data writing to SSD or streaming

With that software one could also build a multi-camera solution for any machine vision or industrial cameras with image processing on NVIDIA GPU. In the simplest case, user can run several processes (one per camera) at the same time to accomplish the task. In more sofisticated approach it would be better to create one image loader which will collect frames from different cameras for further processing on GPU.

There is also an opportunity to utilize different compression options on GPU at the end of the pipeline. We can utilize JPEG (MJPEG), JPEG2000 (MJ2K), H.264 and H.265 encoders on GPU. Please note that H.264/H.265 are implemented via hardware-based NVIDIA NVENC encoder and that compression could be done in parallel with CUDA code.

From the benchmarks on NVIDIA GeForce RTX 2080ti we can see that GPU-based raw image processing is very fast and it could offer very high quality at the same time. The total performane could reach 4 GPix/s. The performance strongly depends on complexity of the pipeline. Multiple GPU solutions could significanly improve the performance.

The software could also work with raw images in PGM format which are stored on external SSD. This is a good method for software evaluation and testing without camera. User can download the source codes from Github or ready binaries for Windows from the download page.

We've done some tests with raw frames in PGM format from bayer Gpixel GMAX 3265 image sensor. That image sensor has resolution 9433 × 7000. For 8-bit mode we've got total processing time around 15 ms which is more than 4 GPix/s. The pipeline included data acquisition, dark frame, FFC, linearization, BPC, white balance, debayer HQLI, gamma sRGB, 16/8-bit transform, JPEG compression (subsampling 4:2:0, quality 90), viewport texture copy, monitor output at 30 fps. The same solution for 12-bit raw image gave us 21 ms processing time.

Here we can see absolutely incredible performance for JPEG encoding on GPU: 65 MPix color image could be compressed within 3.3 ms on NVIDIA GeForce RTX 2080ti.

We recommend to utilize that software as a testing tool to evaluate image quality and performance. User can also test different NVIDIA GPUs to choose the best hardware in terms of price, quality and performance for a particular task.

Custom software design

gpu-camera-sample project is just a simple application which shows how user can quickly integrate Fastvideo SDK info real project with machine vision, industrial and scientific cameras. We can create custom solutions with specified image processing pipeline on NVIDIA GPU (mobile, laptop, desktop, server), which are much more complicated in comparison with that project. We can also build custom GPU-based software to handle multicamera systems for machine vision or industrial applications.

Roadmap for gpu-camera-sample project

  • GPU pipeline for monochrome cameras - in progress
  • H.264/H.265 encoders - in progress
  • Linux version - in progress
  • Resize
  • UnSharp Mask
  • Rotation to an arbitrary angle
  • Support for Imperx, JAI, Baumer, Flir and other cameras
  • Image acquisition and processing for RAW-SDI and HD-SDI cameras
  • JPEG2000 encoder
  • Realtime raw compression (lossless and/or lossy) on GPU
  • Curves and Levels via 1D LUT
  • Color correction with 3x3 matrix
  • Other color spaces
  • 3D LUT for HSV and RGB with cube size 17, 33, 65, 96, 256, etc.
  • Defringe
  • Special version for NVIDIA Jetson hardware (Nano, TX2, Xavier)
  • Interoperability with external FFmpeg and GStreamer

References

Other blog posts on the subject

Contact Form

This form collects your name and email. Check out our Privacy Policy on how we protect and manage your personal data.