Image Processing Framework on Jetson

Author: Fyodor Serzhenko

Nowadays quite a lot of tasks for image sensors and camera applications are solved with the centralized computing architecture for image processing. Just a minor part of image processing features is implemented on the image sensor itself, so all the rest is done on CPU/GPU/DSP/FPGA which could reside very close to the image sensor. The latest achievements at hardware and software solutions allow us to get enhanced performance of computations and to enlarge the scope of tasks to be solved.

From that point of view, NVIDIA Jetson series is suited exactly for the task of high performance image processing from RAW to YUV. Image sensor or camera module can be connected directly to any Jetson via MIPI SCI-2 (2lane/4lane), USB3 or PCIe interfaces. Jetson could offer high performance computations either on ISP or on GPU. Below we show what could be done on GPU. We believe that raw image processing on GPU can offer more flexibility, better performance, quality and ease of management in comparison with hardware-based ISP for many applications.

image processing framework jetson

What is Image Processing Framework for Jetson?

To get high quality and max performance at image processing tasks on Jetson, we've implemented a GPU-based SDK for raw processing. Now we are expanding that approach by creating an effective framework to control all system components, including hardware and software. For example, it means that image sensor control should be included in the workflow at realtime to become a part of general control algorithm.

Image processing framework components

  • Image Sensor Control (exposure, gain, awb)
  • Image Capture (driver, hardware/software interface, latency, zero-copy)
  • RAW Image Processing Pipeline (full raw to rgb workflow)
  • Image Enhancement
  • Image/Video Encoding (JPEG/J2K and H.264/H.265)
  • Compatibility with third-party libraries for image processing, ML, AI, etc.
  • Image Display Processing (smooth player, OpenGL/CUDA interoperability)
  • Image/Video Streaming (including interoperability with FFmpeg and GStreamer)
  • Image/Video Storage

Additional features for the framework

  • Image Sensor and Lens Calibration
  • Quality Control for Image/Video Processing
  • CPU/GPU/SSD balance load, performance optimization, profiling

Implementation of image sensor control at the workflow brings us additional features which are essential. For example, integrated exposure and gain control will allow to get better quality in the case of varying illumination. Apart from that, calibration data usually depend on exposure/gain and it means that we will be able to utilize correct processing parameters at any moment for any viewing conditions.

In general, standard RAW concept eventually lacks internal camera parameters and full calibration data. We could solve that problem by including image sensor control both in calibration and image processing. We can utilize image sensor abstraction layer to take into account full metadata for each frame.

Such a solution depends on utilized image sensor and task to be solved, so we can configure and optimize the Image Processing Framework for a particular image sensor from SONY, Gpixel, CMOSIS image sensors. These solutions on Jetson have already been implemented by teams of Fastvideo and MRTech.

Integrated Image Sensor Control

  • Exposure time
  • Gain
  • AWB
  • ROI (region of interest)

Full image sensor control also includes bit depth, FPS (frames per second), raw image format, bit packing, mode of operation, etc.

GPU image processing modules on Jetson for 16/32-bit pipeline

  • Raw image acquisition from image sensor via MIPI/USB3/PCIe interfaces
  • Frame unpacking
  • Raw image linearization
  • Dark frame subtraction
  • Flat field correction
  • Dynamic bad pixel removal
  • White balance
  • RAW and RGB histograms as an indication to control image sensor exposure time
  • Demosaicing with L7, DFPD, MG algorithms
  • Color correction
  • Denoising with wavelets
  • Color space and format conversions
  • Curves and Levels
  • Flip/Flop, Rotation to 90/180/270 or to arbitrary angle
  • Crop and Resize (upscale and downscale)
  • Undistortion via Remap
  • Local contrast
  • Tone mapping
  • Gamma
  • Realtime output via OpenGL
  • Trace module for debugging and bug fixing
  • Stream-per-thread support for better performance
  • Additional modules: tile support, image split into separate planes, RGB to Gray transform, defringe, etc.
  • Time measurements for all SDK modules

Image/Video Encoding modules on GPU

  • RAW Bayer encoding
  • JPEG encoding (visually lossless image compression with 8-bit or 12-bit per channel)
  • JPEG2000 encoding (lossy and lossless image compression with 8-16 bits per channel)
  • H264 encoder/decoder, streaming, integration with FFmpeg (8-bit per channel)
  • H265 encoder/decoder, streaming, integration with FFmpeg (10-bit per channel)

Is it better or faster than NVIDIA ISP for Jetson?

There are a lot of situations where we can say YES to this question. NVIDIA ISP for Jetson is a great product, it's free, versatile, reliable, and it takes less power/load from Jetson, but we have our own advantages which are also of great importance for our customers:

We've built that software from the scratch and we've been working in that field more than 10 years, so we have an experience and we can offer reliable solutions and support. Apart from that we are offering custom software design to solve almost any problem in a timely manner.

What are benefits of that approach?

That approach allows us to create embedded image processing solutions on Jetson with high quality, exceptional performance, low latency and full image sensor control. Software-based solution in combination with GPU image processing on NVIDIA Jetson could help our customers to create their imaging products with minimum efforts and maximum quality and performance.

Other blog posts about Jetson hardware and software

Contact Form

This form collects your name and email. Check out our Privacy Policy on how we protect and manage your personal data.