Dynamic bad pixel correction in bayer images on GPU
Updated:
Fastvideo bad pixel correction detects and repairs defective sensor pixels on the GPU as part of the RAW pipeline, using dynamic and static defect maps. A 12 MPix 12-bit frame is corrected in under ~1 ms, so it runs inline in real-time camera processing without slowing the stream.
Detailed benchmark data. Measured results are published on our benchmark pages: all benchmarks.
- Dead pixels don't respond to incident light (usually output constant value)
- Hot pixels generate maximum values for incoming light (usually output is significantly higher than it should be)
- Bright pixels produce exaggerated values for incoming light (usually the output is slightly higher than it should be)
- Black pixels - these are pixels with a weak response (they usually output less value than they should)
If we look at the detailed specifications of any image sensor from any manufacturer, we will see a confidential document describing bad pixels and their locations. Yes, every image sensor manufacturer knows that there are some bad pixels and that the number of bad pixels will increase as the sensor is used. In order to consider a particular image sensor as good, we could compare the actual positions and number of bad pixels with the specification. Usually, bad pixels are not allowed to be near the centre of the image sensor, but closer to the edges.
Camera manufacturers usually sort image sensors before making cameras, because different camera applications do not have the same image quality requirements. If we compare a surveillance camera with a microscope camera, they simply can't have the same number of bad pixels.
Image sensor manufacturers rate their devices according to the number and type of defects present.
Bad pixels are really bad if we don't remove them before image segmentation, image recognition, stitching, 3D modelling, etc. These bad pixels aren't good for viewing either. So we see too many reasons to suppress them.
Once bad pixels are fairly common, there are several ways to remove them:
- We could fix them inside the camera on the FPGA according to the known list of coordinates
- They could be fixed in RAW data in the software
- Bad pixels in RGB data could be fixed in software
Bad pixel suppression in the FPGA is a very convenient way and it's super fast, but we need to keep the list of bad pixels up to date. To check the appearance of new bad pixels, we need to run calibration tests to update this list. Unfortunately, this is not always convenient to do, so this approach is not 100% applicable.
The most flexible approach is to remove bad pixels from RAW data using software. We can either interpolate bad pixels according to a known list of coordinates, or we can implement a special dynamic filter that is able to detect and interpolate bad pixels in RAW data.
If we haven't removed bad pixels from RAW, we can still remove them from RGB. In such a case, removal is still possible, but it's done after demosaicing, which is why some "traces" of bad pixels might be found on the neighbouring pixels. This method is also possible, but it's not the best.
Bad pixel correction filter
What is bad pixel correction and why is it needed?
We've implemented a powerful filter for Bad Pixel Correction (BPC). Basically, we need to remove impulse noise, and such removal is almost always good for the image. The main idea is not to apply this filter to all pixels, but only to those that look like bad/hot/dead pixels. We analyse the environment of each RAW pixel and formulate some criteria to recognise the pixel as bad. This is done in the RAW domain, so we apply different criteria for green and red/blue pixels. The algorithm can work over a wide range of illumination levels and we've done some tests to investigate these situations.
We see almost the same situation when we calibrate a camera to get a Flat-Field Correction (FFC) image. Hot pixels can affect the results of FFC, so we need to interpolate them before we do any calculations.
Apart from repairing bad pixels, we should always keep an eye on the total number of pixels processed by this filter. This is an indirect way of checking image sensor degradation and quality. This solution is practical and could be applied to RAW data from any camera. Soon we will extend this filter to work with RGB data as well, because there is such a need to fix bad RBG pixels.
Bad pixel replacement on NVIDIA GPU
We've implemented the BPC filter on the GPU to provide good performance and quality even for cameras with high resolution image sensors in real-time applications. We process bad pixels in the RAW domain and we can do it really fast. A frame from a 12 MPix 12-bit image sensor could be fixed in less than ~1 ms on NVIDIA GeForce GTX 1080 even without having all the coordinates of the bad pixels - we detect and interpolate bad pixels in real time. This is actually a dynamic bad pixel correction algorithm. This feature is now part of the Fastvideo SDK and it's available for NVIDIA mobile, laptop, desktop and server GPUs.
- Input data: 8/10/12/14/16-bit RAW (bayer or grayscale) from HDD/SSD/RAID in PGM/DNG/CinemaDNG format
- Output format: 8/12/16-bit RAW with corrected pixels
- Maximum image resolution up to 16k×16k pixels and more
- OS support: Windows-10/11, Linux Ubuntu, and L4T for Jetson
To check performance and quality for BPC algorithms, you can download Fast CinemaDNG Processor software for evaluation.
About the author
Fyodor Serzhenko, PhD, is the founder and CEO of Fastvideo. He earned his PhD at the Moscow Institute of Physics and Technology (MIPT) in 1993. Since 2009 he has led the development of Fastvideo’s GPU-accelerated image codecs and ISP modules, including the Fastvideo SDK. Connect on LinkedIn.
Why you can trust these results
Fastvideo has built GPU-accelerated image processing software since 2009. Our codecs comply with the standards they implement — JPEG (ITU-T T.81 / ISO IEC 10918) and JPEG2000 (ITU-T T.800 / ISO IEC 15444) — and our tools are open source on GitHub. Every performance figure we publish is reproducible: see our benchmark methodology, download the Fastvideo SDK benchmark report (PDF), and measure it on your own GPU.