FastVCR software for MIPI CSI2 cameras: realtime processing on NVIDIA Jetson
In today's technologically advanced world, capturing high-quality images and videos has become an integral part of various industries. One of the key components responsible for this remarkable feat is the MIPI CSI2 camera interface. MIPI CSI2 (Mobile Industry Processor Interface Camera Serial Interface 2) is a widely adopted camera interface standard designed specifically for mobile devices, but it has found its way into other applications as well. It enables seamless communication between image sensors and application processors, ensuring efficient data transfer and high-quality image capture.
MIPI CSI2 cameras offer several notable features and benefits
MIPI CSI2 cameras find applications in various industries
MIPI CSI2 cameras have revolutionized the way we capture and process images and videos. With their high data transfer speed, low power consumption, and compact size, they have become an integral part of various industries, ranging from mobile devices to automotive and medical applications. As technology continues to advance, we can expect further enhancements in MIPI CSI2 cameras, enabling even more impressive imaging capabilities in the future.
MIPI CSI2 cameras and Jetson
By leveraging the capabilities of NVIDIA Jetson platforms and integrating them with MIPI cameras, developers can unlock a whole new level of possibilities. These combinations empower devices to perform complex computer vision tasks, including object detection, tracking, facial recognition, and much more. The real-time processing capabilities of Jetson, combined with the high-quality imaging from MIPI cameras, create a potent solution for a wide range of applications.
The Jetson platform provides developers with an extensive set of tools and libraries to develop and optimize software for Jetson devices. It includes NVIDIA's deep learning framework, CUDA, which allows developers to leverage the GPU's parallel processing capabilities to accelerate AI inferencing.
Software for MIPI CSI2 cameras
In most cases MIPI cameras are connected to NVIDIA Jetson modules and the software for processing is based on libargus library. This is the way of utilizing hardware-based ISP which is available at each Jetson. That approach has important limitations which could be overcome with GPU-based processing on Jetson. We've implemented our own ISP pipeline on CUDA and user can tune it accoding to the task to be solved.
To fully utilize the potential of MIPI cameras, we've extended our FastVCR software to take advantage of the Jetson hardware for various camera applications. This is a key approach to utilize the power of Jetson's GPU to perform real-time image processing tasks on MIPI cameras.
To work with a particular MIPI camera, we need a driver which is offering access to RAW frames and various image sensor modes of operation. This is not always the case, but it's a mandatory requirement.
Why do we need high performance software for MIPI cameras?
If we have a look at MIPI camera with IMX477 image sensor (12 MPix / 12-bit / 30 fps), we can see that real time processing performance at maximim resolution, max bit depth and at 30 fps is not a simple task. Currently it can be solved either with libargus library or with GPU-based SDK on CUDA, because camera bandwidth is quite high.
Modern MIPI cameras could offer both high resolution and high fps, so it's quite a complicated task to perform high quality real time processing without dropped frames. That is exactly what we can do with FastVCR software which is running on NVIDIA Jetson. In our pipeline we've included high quality demosaicing algorithms together with other image processing modules to take into account color science and high performance requirements.
FastVCR software for MIPI CSI2 camera control
16-bit ISP for MIPI CSI2 cameras on GPU
FastVCR performance benchmarks
That software was built with the idea of GPU image processing to be capable of working at high performance camera applications and for multicamera systems. FastVCR software can process RAW images in real time from 12-MPix MIPI camera with IMX477 image sensor (4056×3040, 12-bit, 30 fps) on the NVIDIA Jetson Orin, the pipeline includes RAW import, RAW-to-RGB transform, color correction, JPEG encoding, storage to SSD. Actually, on that hardware we need around 20 msec to process one frame on the GPU, which could be an indicator of maximum performance.
Optional image processing modules on GPU
FastVCR command-line application
Quite often we need to run a software without any GUI, and it could happen in different situations. This is the case for drone applications or for any other task with remote camera control. This is also the case for any long-term unsupervised video recording and streaming.
To meet these requirements, we've developed a CLI application which has all the above features of FastVCR software and it could work without GUI. We are still able to have full control on image sensor and image processing parameters at real time. For video preview we supply our own player with RTSP client or one can use VLC instead.
All the above features are designed on the basis of our CUDA image processing SDK. Now our customers have an opportunity to utilize GPU-accelerated software components in their applications as a part of video processing pipeline.
Apart from FastVCR software, we are offering custom software design to build a solution according to customers' requirements. We can implement application-specific modes of operation to solve the task you need. Please fill the contact form below with your project info and send it to us.
Blog posts on the subject