Multi Camera Systems with GPU Image Processing
Multi-camera system can do the job which is impossible for a single camera. This is the way to get bigger field of view, higher resolution, better data rate, different spectral sensitivity, etc. Nowadays multi-camera systems are utilized in many industries for various applications. Such a system is able to provide much more information in comparison with a single camera. Nevertheless, multicamera systems are quite complicated to handle, and we need to take into account quite a lot to make them working properly and at full speed.
Fig.1. Full view of multi-camera rig from Infinite Realities company
Applications for multi-camera systems
- VR and Digital Cinema: Creation of 3D models is usually done via multicamera systems. 3D computer-animated movies is a current trend in digital cinema. The art of digital visual effects has matured to the point where it’s possible to recreate almost anything. Precision of 3D model and character animation depends on the number of cameras, their resolution, bit depth, etc. 3D modelling software relies on image series from modern multiple camera systems.
- Medical imaging: Various imaging systems such as endoscopes, microscopes, X-ray devices or operating room monitoring cameras are used in medical applications. A major challenge is to control all these video sources in the clinical environment, to provide the most diverse cable concepts, operate the cameras, transport, transform, display, document and archive images and video.
- Mapping: there is a task of multi-camera tracking and mapping with non-overlapping fields of view (real-time pose estimation systems). Multicam camera cluster could be constructed by using multiple perspective cameras mounted on a multirotor aerial vehicle and augmented with tracking markers to collect high-precision ground-truth motion measurements from an optical indoor positioning system.
- Video surveillance task includes multi-camera calibration, computing the topology of camera networks, multi-camera tracking, object re-identification, multi-camera activity analysis and cooperative video surveillance both with active and static cameras. With the development of surveillance multi-camera systems, the scales and complexities of camera networks are increasing and the monitored environments are becoming much more complicated.
- Motion tracking: multi-target tracking (MTT) and multi-object tracking (MOT) are active and challenging research topics to increase the robustness of tracking. Typical approaches for multicamera tracking assume overlapping cameras observing the same 3D scene, exploiting several real-world constraints like a common geometry. Most multiple camera approaches for multiple object tracking rely on background subtraction and project all foreground pixels onto a common ground plane.
- Sport: high-end multi-camera broadcasting (live multi-cam streaming) for sport competitions is a global trend. Apart from that, realtime sports video analytics, replay and in-depth event sports coverage could be much more effective with multicamera systems.
- 3D scanning and photogrammetry: modern state-of-the-art photogrammetric 3D/4D scanner systems are based on multi-camera technology. Such multicamera rigs enable rapid capturing of 2D images for creating 3D models of still or moving objects, animals and humans.
- Robotics, UAV, self-driving units need precise info about their own position and positions of other surrounding objects which could be measured with multiple camera systems. Multi-camera system could cover full 360-degree field-of-view around the car to avoid blind spots which can otherwise lead to accidents. Standard image processing pipeline for 3D mapping, visual localization, obstacle detection, need to be changed to take full advantage of the availability of multiple cameras instead of treating each camera individually.
- Machine vision and industrial cameras are utilized for sofisticated control in production. Such control and automation tasks just can't be solved with one camera, so multi-camera systems are necessary.
- High speed imaging also benefits from multiple camera systems. Recordings from different angles at the same time offer much more information for scientific research, testing, diagnostics.
Architecture of multi-camera systems
To build a multi-camera system, we need to take into account many aspects of future imaging solution. We could briefly describe the main ideas which are really important to create viable multicamera system.
Cameras and accessories for multi-camera system
- Image sensors: mono/color, NIR, spectral sensitivity, dimensions, pixel size, bit depth, resolution, etc.
- Additional sensors: lidar, GPS
- Illumination: laser, strobe, synchro, structured/modulated, wavelength-specific
- Trigger: master, master-slave, jitter
- Optics: mount, quality, focal lengths, filters, remote control, etc.
- Other constraints: size, weight, heat, temperature, power consumption
- PCIe switch: upstream PCIe x8 Gen3, downstream PCIe or USB, trigger & power
- Availability of interfaces: PCIe, USB3, GigE / Gig10E, CameraLink, Coax, etc.
- Cable connections: Flat ribbon, iPass, FireFly
- Cable length, ability for cable replication
- Power: external power supply, battery
Software for multi-camera systems
- Multi-camera control software
- Realtime data acquisition and recording
- Image processing pipeline for each camera
- Realtime or offline image processing on CPU or GPU
- Output compression
- Storage or streaming
- Post processing
Multi camera system configurations
- Inside In (180/360 panorama systems)
- Inside Out (3D Dome)
- Camera array
- Cluster of cameras (set of cameras at one box/body)
Fig.2. 3D scanning multi-camera system from Infinite Realities company
Input calibration parameters for each camera
- Dark frame
- Flat-Field frame for FFC
- Digital Camera Profile (DCP)
- Lens Correction Profile (LCP)
- Camera intrinsic parameters: principal point, aspect ratio, focal length, distortion center and coefficients
- Camera extrincis parameters: translation, rotation, homography
Fig.3. Embedded multi camera system on NVIDIA Jetson from XIMEA company
Image processing pipeline for multi-camera system usually is essentially the same as for a single camera. The difference comes from synchronization, relative positions, individual camera parameters and from high data rate. The most difficult problem is high data rate. When the amount of frames per second is increased in many times, we just can't scale CPU performance accordingly at the same hardware, as soon as even for a single camera we have quite complicated multithreaded software for image processing on CPU. Further scaling is not possible in many cases, so we need to implement another approach.
Now we can see that performance boost with GPU for image processing in camera applications could be a good solution to solve the matter. We do have excellent benchmarks for GPU-based image processing, and this is the way to scale the performance for multi-camera system. On GPU we can process raw streams from multiple cameras in realtime. We just need to supply individual calibration info for each frame to let GPU know corresponding parameters for each camera.
We've implemented quite a lot of such solutions for different camera applications. Our experience helps our customers with their projects. We have GPU-based Image & Video Processing SDK which is capable to solve such multicamera tasks.
Fig.4. Fastvideo SDK to process raw images on GPU for multi-camera systems
Pipeline example for multicam system on GPU
- RAW iImage acquisition data import to GPU memory
- Dark frame subtraction
- Flat-Field Correction
- White Balance
- Exposure correction
- RAW 1D LUT (composite or per-channel)
- Color correction with matrix profile
- Curves and levels for RGB/HSV
- Projection together with undistortion
- 3D LUT (RGB/HSV)
- Crop and Resize
- Gamma with output color space conversion
- Compression to JPEG, JPEG2000, TIFF, PNG, EXR, H.264, H.265
- Data storage or transmission
Please send us your request concerning your anticipated image processing pipeline for your multi-camera system. To evaluate the quality and the performance of our solution, please download Fast CinemaDNG Processor software for Windows. At the same link you can download DNG image series for testing as well.
Fig.5. Full image processing pipeline on NVIDIA CUDA for each camera in multi-camera system
The above image processing pipeline for NVIDIA GPUs is capable to solve standard task of RAW transform to RGB (compressed or uncompressed) for off-line and realtime applications. We've utilized it for different multi-camera systems and in all cases we've got very good results. GPU is a powerful means to do realtime image processing for multicamera systems and we suggest to try our SDK for these applications.