RawZipper - RAW Bayer Codec on NVIDIA GPUA multitude of image processing tasks in camera applications necessitate the storage of RAW frames without conversion to RGB. This conversion can be conducted subsequently or offline. This assertion is applicable to a multitude of applications, including, but not limited to, street view, aerial imaging, and aerospace applications. Consequently, the RawZipper (RAW Bayer Codec) was developed to compress RAW data prior to its storage on an SSD. The RawZipper Codec has been implemented for the purpose of compressing and storing raw image data captured by monochrome and bayer cameras. Bayer data adheres to a Bayer pattern (RGGB, BGGR, GRBG, or GBRG), in which each pixel captures only one color channel. True lossless or lossy (nearly or visually lossless) compression could be employed to preserve the original data for subsequent raw processing. Applications for RawZipper
In select circumstances, the implementation of lossless encoding on the captured RAW images is a viable option. However, the compression ratio (CR) of these algorithms is typically around 2:1, which is not particularly high. Notwithstanding, this is a mathematically lossless algorithm, meaning that each image will be restored exactly as it was right after acquisition (before compression). The following lossless algorithms for image compression could be employed: Lossless JPEG (1–16 bits), JPEG-LS (1–16 bits), JPEG 2000 (up to 16 bits), JPEG XS (up to 16 bits), JPEG XL (up to 16 bits), etc. In certain cases, the application of lossless compression is mandatory, particularly within the context of scientific research. There are also some options to use lossless video codecs. Video codecs are usually offering much higher compression ratio in comparison with the results for still image compression. The lossless H.265 encoding algorithm can work with color video streams, but it's only applicable to 8/10-bit color (NV12, YUV) output, not to RAW. The same is true for the lossless AV1 codec, which can encode up to 10-bit YUV (4:2:0, 4:2:2, 4:4:4), but not RAW. These methods aren't applicable for the most common use case, which is raw image compression from Bayer or monochrome image sensors with bits per pixel of 12 or higher, up to 16 bits. However, there are cases where video codecs could be a suitable solution for such a task. While the application of lossy encoding to RAW images is indeed logical in order to achieve a significantly superior CR, it is imperative to exercise control over the degradation of image quality, a prerequisite for the efficacy of such algorithms. Minor distortions in the RAW-encoded frames have the potential to result in significant artifacts at the processed RGB images. However, these artifacts would only become apparent after the conversion from RAW to RGB. This approach has the potential to achieve a CR that exceeds 2:1, necessitating the identification of an optimized solution that considers both the requirements for image quality and the compression ratio. It's worth mentioning that video codecs like HEVC look attractive for lossy RAW image compression after some preprocessing because they are hardware-based at NVIDIA GPUs, so they don't influence on GPU load too much. But their performance is not high enough to deal with real time data streams from modern cameras, though in some casese it could be a viable solution. RAW encoding algorithms of RawZipper
Benefits of RAW Compression
Key Features of RawZipper
We can integrate RawZipper into your imaging system to perform the whole job fully on GPU. Please check the description of our GPU Image & Video Processing SDK to evaluate what we can do on GPU. Image Quality and CR ConsiderationsIn the context of RAW bayer frames derived from conventional 12-bit image sensors manufactured by Sony, Gpixel, On-Semi, and CMOSIS, a compression ratio of CR~2 is typically attained for lossless algorithms. The same concerns 14-bit raw data from photo cameras of SONY, Canon, Nikon, etc. This is due to the fact that useful data also includes noise, which could be important in scientific applications that require lossless data compression. Lossless encoding demonstrates lower performance in comparison to lossy codecs. The application of lossy encoding typically results in a trade-off between loss and enhanced performance, the efficacy of which is contingent upon the content of the image, the parameters employed, and the hardware utilized. It is imperative to understand the mechanisms of controlling the level of loss in order to ensure the preservation of high-quality images for a specific task. In most cases, it is possible to encode RAW frames at a compression ratio (CR) of 5–10 times, thereby achieving a peak signal-to-noise ratio (PSNR) for 12-bit bayer images of 50–55 decibels (dB) or better. The PSNR is computed for the original RAW Bayer image and the same image after processing, encoding, decoding, and restoring. It is acknowledged that PSNR metrics are of limited utility. To this end, visual image quality assessment is performed to determine the most suitable default processing parameters for a given project. A visual comparison was conducted, not between RAW images, but between RGB images that had undergone processing with identical parameters. For instance, there should be no perceptible differences between the viewing source and the processed images when viewed at a 400%-500% zoom level. For applications necessitating maximum image quality, a CR range of 4-5 could be employed, thereby attaining PSNR levels for RAW images exceeding 60 dB. Numerous cases exist that exhibit significantly weaker image quality requirements and a pronounced need for enhanced CR. Achieving a CR of up to 20 for such applications is a feasible objective. It is imperative to acknowledge that the RAW Bayer Codec with lossy encoding is not merely a codec in the conventional sense. This solution is characterized by its sophistication and the extensive range of its capabilities, which extend well beyond data compression and decompression. The proposed methodology strikes a balance between preserving raw data and performing additional image processing to improve contrast-to-noise ratio and minimize image distortion. Conventional wisdom suggests that the quality of RAW bayer images is significantly enhanced by the implementation of advanced technology, as evidenced by the final superior color reproduction and image quality that is achieved in comparison to conventional JPEG and JPEG 2000 codecs. Once the requirements of a given project are thoroughly understood, the software can be customized to optimize results for specific applications. Image quality control for RAW CodecThe RawZipper Codec can work both in lossless and lossy modes. When we're working in lossy mode, we end up with some image distortions. It's important to measure the level of losses and to evaluate them visually and numerically. We've got to check both the signal and the noise to make sure we can reproduce the original image with high precision. It's an imperative to restore both the signal and the noise at the same time. So, we've come up with these tests and numbers:
For evaluation we've taken a raw frame with ColorChecker from the DALSA/FLIR camera (20 MPix, 4512 x 4512, 12-bit, RGGB). We've applied lossy raw bayer encoding with CR = 10 and got PSNR = 45.3 dB. The standard deviation (σ) has changed at average less than 2%, and the SNR has changed less than 0.15 dB. We've taken measurements for each patch of the ColorChecker. RawZipper performance benchmarks for lossy RAW Bayer encoding on GPURealtime RAW Bayer encoding with lossy algorithm is very important for many applications and it's the most demanding part of the codec, because usually decoding could be done offline. Full set of benchmarks for RawZipper Codec will be published later, and here we present some preliminary results. For 12-bit RAW Bayer images with resolution 12 MPix, we can perform lossy encoding with CR around 10 with the performance up to ~1 GPix/s on the NVIDIA GeForce RTX 4090 and we have fair chances to accelerate that solution soon. GPU pipeline for RAW decoding, processing and visualization
The aforementioned pipeline is computationally demanding, and attaining high performance for the task is challenging. In order to facilitate the visualization process, it is imperative to attain a frame rate of at least 60 frames per second (fps). For the majority of resolutions, this objective can be achieved in real time using an NVIDIA GPU. We utilize our GPU Image Processing SDK for the above pipeline and it can handle data rates up to 2-4 GPix/s on GeForce RTX 4090. Such a solution is available for all NVIDIA GPUs: mobile, laptop, desktop, server. Useful links on the subject |