FFmpeg Remap Filter on GPU

Remap filter is intended to copy a source image to a target image according to two maps (ymap/xmap) which are usually supplied in two files. Map files are passed as a parameters and they are usually in binary PGM format, where the values are y(rows) and x(columns) coordinates of the source frame. The target image dimension is based on mapfile dimensions which are specified in the header of the mapfile. Dimensions of ymap and xmap must be equal, data values must be positive or zero. These coordinates are in floating point format, so fraction pixel positions can be used.

ffmpeg remap filter gpu

Remap chooses source pixels by using pixel coordinates explicitely supplied in two 2D device memory image arrays pointed to by the pXMap and pYMap pointers. The pXMap array contains the X coordinate and the pYMap array contains the Y coordinate of the corresponding source image pixel to use as input.

In a standard FFmpeg we can find several filters for remapping: v360 (11.213) and remap (11.164). Usually Remap Filter on CPU is quite slow an it could be easily checked by benchmarking of these filters with FFmpeg. Nevertheless, remap filter could be performed much faster on GPU. We've implemented that filter on NVIDIA GPU with Fastvideo SDK and it's working very fast even on laptop GPUs. On professional GPUs we can decode, remap and encode multiple video streams with high performance and minimum latency.

How to remap video on GPU via FFmpeg

The task of video remapping on GPU is quite standard and it could be utilized to create 360 movies with equirectangular projection. We can run our GPU-based FFmpeg filter to remap video file or to apply it to available video stream, which is also possible due to high performance of that algorithm on GPU. Below you can see pipeline description for such a task of video file remapping.

  • Open input MP4 file with FFmpeg
  • MP4 file parsing with FFmpeg
  • Send data for further processing on GPU
  • MP4 decoding via NVIDIA NVDEC
  • Transform of output NV12 format to RGB
  • Remap
  • Transform to NV12
  • Output encoding via NVIDIA NVENC
  • FFmpeg stores encoded frames to MP4 container

Example command for FFmpeg Remap Filter

Command line below shows how to run FFmpeg Remap filter on GPU for h264-encoded video in mp4 container:

ffmpeg -y -vsync 0 -hwaccel cuvid -c:v h264_cuvid -i source.mp4 -vf remap_fastvideo=mapX=x.pgm:mapY=y.pgm -c:a copy -c:v h264_nvenc -b:v 5M target.remap.mp4

Why we can remap mp4 much faster than standard FFmpeg?

  • We utilize FFmpeg only for data parsing and for data packing to MP4 container
  • Video decoding is done via NVIDIA NVDEC (on GPU)
  • All image format transforms are done on GPU
  • Remap is applied on GPU as well
  • Video encoding via NVIDIA NVENC (on GPU)
  • That FFmpeg Remap Filter is based on highly optimized Fastvideo SDK for image and video processing on GPU

It's worth mentioning that any stage of the above image processing pipeline on CPU instead of GPU will bring us slower results and it makes sense to implement full pipeline on GPU. That's exactly what we've done to speedup FFmpeg Remap Filter on GPU.

Benchmarks for FFmpeg Remap Filter on GPU

We've done several tests for remapping of live video which was originally compressed with h.264 (mp4 format). For a video with frame resolution around 5 MPix we've got the performance 160 fps on NVIDIA GeForce RTX 2070S GPU, though on Intel Core i9-9960X CPU we've got much less for the same test. This is just an example to show how fast FFmpeg could be for video decoding, remapping and encoding on desktop NVIDIA GPU. We can actually do the same on any NVIDIA GPU: mobile, laptop, desktop, server and for multiple GPU setup.

As soon as NVENC and NVDEC are hardware-based encoder and decoder, then video encoding and decoding are performed on NVIDIA GPU in parallel with CUDA code. This is very important to get additional acceleration for GPU-based remap filter. Actually, this is the way to create even more complicated remap filter by adding additional image processing features on GPU to the pipeline.

We would suggest to try Fastvideo SDK with sample application for FFmpeg Remap Filter to check the performance and quality for your video on your GPU. That filter is available for Windows, Linux and L4T.

We've also implemented GPU-based remap feature at Fast CinemaDNG Processor software to solve various tasks of lens undistortion and image rotation by an arbitrary angle for digital cinema applications. In that software there is a benchmark window and user can see how much time it's necessary for particular Remap transform on CUDA.

Applications for FFmpeg RemapFilter

  • Remap to convert a circular fisheye or dual-fisheye video to equirectangular
  • Convert a fisheye image into a panoramic, spherical or perspective projection
  • Radial and tangential undistortion
  • Chromatic aberration removal
  • Image rotation by an arbitrary angle
  • Dewarping for cylinder or spherical projections
  • Affine and Perspective transforms
  • Arbitrary projections via 2D maps

Standard Remap Procedure

To apply FFmpeg RemapFilter, user needs to create two maps first. This is usually done via calibration.

  • Capture set of images with chess pattern for calibration purposes
  • Calculate set of coefficients and image center position
  • Calculate offsets from expected positions (with predefined steps for X and Y)
  • Create two maps: one with X offsets and the other one with Y offsets
  • Remap can directly interpret these two offset tables and change the pixel positions accordingly
  • All new pixels will be interpolated via linear, bicubic or lanczos algorithm

Contact Form

This form collects your name and email. Check out our Privacy Policy on how we protect and manage your personal data.