RAW Processing for Aerial Imaging
Updated:
Author: Fyodor Serzhenko
Tags: RAW Processing, Aerial Imaging Workflow, CUDA, GPU
RAW processing gives aerial imaging and photogrammetry better final quality than in-camera JPG: capturing RAW and applying per-frame exposure correction on the GPU keeps brightness and color consistent across a flight, which is exactly what stitching and orthomosaic work need. This article walks through a practical GPU RAW workflow for aerial and drone imaging — from SSD import and culling to orthomosaic color correction — using Fast CinemaDNG Processor for RAW and Agisoft Metashape for photogrammetry. For the product overview, see aerial imaging software.
While working on aerial imaging projects, it's usually multi-stage image processing workflow which requires different software solutions. And here we discuss a general approach for image processing, and particularly the role of RAW data and RAW processing to get a better quality at the final result.
We will not consider in detail the stage of flight planning and image capture, so we will start from available RAW/JPG data set on SSD which is acquired during the flight.
Brief Overview for Aerial Imaging Workflow
- Data import from SSD, create reserve copies, fill project metadata
- RAW/JPG preview and culling
- Conversion of RAW to 16-bit TIFF
- RTK/PPK
- Photogrammetry: align photos, build dense cloud, build mesh, build DEM, build Orthomosaic
- Radiometry correction: Flat-Field Correction, Undistortion, Radiometry calibration
- Mosaic and Seamlines
- Final color correction and toning: brightness/color alignment, curves/levels, local contrast, shadow removal
- EPSG:326XX for UTM, LZW compression, tiling (256x256), export to GeoTIFF, vector mask (Shapefile)
- Export (Orthomosaic, DEM, 3D model) and archiving (RAW and Metashape project)
Here we will consider RAW image processing, so it's mostly connected with stages 2 and 3, though it could be also taken into account at 6, 7 and 8. For processing we will utilize Fast CinemaDNG Processor software for RAW image processing and Agisoft Metashape for photogrammetry tasks.
For the beginning we need to understand why do we need RAW image processing? Cameras are usually capable of capturing images both in RAW and/or JPG format. It's worth mentioning that JPG files have less size, so it is very promising to use them to avoid running out of storage during the flight.
Still, the idea of using JPG compression has negative consequences as well. For example, any problem with setting of correct image sensor exposure will bring us either dark frame or overshoot. This is the case even for well-known camera brands, and this is what we get quite often. Both cases are not good, but from a dark frame we can usually get a decent image by applying exposure correction at the software, though in the case with overshoot it's not possible to restore the image content. This is the risk to get a non-complete data for mapping and it can cause reflight, which is not desirable.
We suggest running the following workflow to avoid such problems:
- Use RAW image capture (don't use JPG)
- Apply individual exposure correction to each frame at RAW-to-RGB conversion to get the same image brightness for all frames in each image series
This is possible with Fast CinemaDNG Processor software and it's very fast because all processing is done on the GPU. Average performance on a good GPU could be in the range of 1-2 GPix/s, so we can also preview the result of processing as video to check brightness consistency for all frames in the set. We can apply culling at that stage as well, if we see some images which have to be removed from the set. Flat-Field correction could be done here to get high quality images which are ready for stitching and other processing with Agisoft software.
It's worth mentioning that sharpening, undistortion, radiometric correction, and final toning and color correction should not be applied at that stage of processing to avoid problems with stitching. They are usually applied later at the workflow.
Following these procedures, we can get good JPG images for each image series with consistent color/brightness, but we still can run into the problem with Orthomosaic color mismatch. That could happen quite often because there were different weather conditions at each flight. Other obstacles like clouds and haze could also make color/brightness different. This is why we can get a big map which consists of regions with different color/brightness, and this is not good. We need to apply additional procedures to make colors compatible across different image sets at Orthomosaic.
Orthomosaic correction
We need to start from the specification of initial data set to cope with that task:
- JPG images with consistent color/brightness for each flight
- Ortho images after projection for each flight
- Additional info about positioning of each ortho image for each flight
We create a window to draw all ortho images there and to evaluate visual mismatch for color and brightness. We also need to be able to select all images from the same flight to be able to see the difference with neighbouring images from different flights.
The software will allow us to apply exposure correction for the whole set of images from a particular flight and to see whether it could improve the visual appearance of the total map. Usually color calibration is almost the same if we compare bright sunshine and cloudy weather, the difference mostly comes from brightness. If this is the case, we will be able to correct brightness with exposure correction and we will keep these values to be able to apply them at the second pass to RAW data. That second pass will bring us JPG images with consistent brightness and color and this is exactly what we need.
Roadmap
- Automatic shading correction without preliminary calibration
- GPU-based ISP optimization
- Benchmarking of CUDA-based aerial image processing
Aerial imaging FAQ
Why use RAW instead of JPG for aerial imaging?
JPG saves storage during the flight, but an exposure error can make a frame unrecoverable and force a reflight. RAW keeps the full data, so you can apply per-frame exposure correction afterwards and keep every frame in a series at the same brightness.
How do you keep brightness and color consistent across a flight?
Capture RAW and apply individual exposure correction to each frame at RAW-to-RGB conversion on the GPU. It runs in the range of 1–2 GPix/s on a good GPU, fast enough to preview the whole set as video and cull frames while checking brightness consistency.
How do you fix orthomosaic color mismatch between flights?
Evaluate the ortho images together, apply exposure correction per flight to match brightness, then re-apply those values to the RAW data in a second pass. The result is JPG images with consistent color and brightness across the whole map.
Which steps run on the GPU?
RAW-to-RGB conversion, exposure and flat-field correction and other heavy stages run on the GPU via Fast CinemaDNG Processor; sharpening, undistortion and final toning are applied later in the workflow to avoid stitching artifacts.
Links
Fastvideo Image and Video Processing SDK for NVIDIA GPUs
High performance denoising algorithms on GPU
GPU tone mapping software with ALTMapper algorithm
GPU RAW image processing for aerial imaging (overview)
RawZipper — RAW image compression
About the author
Fyodor Serzhenko, PhD, is the founder and CEO of Fastvideo. He earned his PhD at the Moscow Institute of Physics and Technology (MIPT) in 1993. Since 2009 he has led the development of Fastvideo’s GPU-accelerated image codecs and ISP modules, including the Fastvideo SDK. Connect on LinkedIn.
Why you can trust these results
Fastvideo has built GPU-accelerated image processing software since 2009. Our codecs comply with the standards they implement — JPEG (ITU-T T.81 / ISO IEC 10918) and JPEG2000 (ITU-T T.800 / ISO IEC 15444) — and our tools are open source on GitHub. Every performance figure we publish is reproducible: see our benchmark methodology, download the Fastvideo SDK benchmark report (PDF), and measure it on your own GPU.