How to break the speed barrier for machine vision image processing?What do we usually overlook to speed up real-time image processing?Machine vision cameras are widely used in industry, science, and robotics. However, when working with them, the same question invariably arises: "How to process the data received?" And that's a really good question. But why does it arise at all? The point is that cameras usually transmit raw data (RAW) at high frame rate, which takes up a lot of memory and needs to be converted to the required image format in realtime. Image processing algorithms must provide the quality and speed necessary for the task at hand. Unfortunately, it is sometimes not easy to ensure both quality and speed at the same time. That’s why, whenever there’s a task which requires processing a lot of images in real time, experts put a high priority on optimizing computer vision-related algorithms. It’s even more important when there’s a limited budget, or the physical size or power consumption of the device is constrained for practical reasons. Generally, high-quality algorithms that perform computations on Intel/AMD processors do well with this task. However, there are special cases: Case 1. Processing of images from high data rate machine vision cameras, which is the case for high image resolution or a high frame rate. Case 2. Multi-camera system with real-time image processing. For such situations, the capabilities of a CPU are not enough. CPU just can't handle the huge data stream quickly enough (for example, when it’s dealing with gigapixels per second), and this leads to the unavoidable loss of some data. Unfortunately, it’s difficult to speed things up further on a CPU without a trade-off for quality. So, how can we speed up image processing without losing quality? The main idea for the solution was to transfer most of the computations from the central processor (CPU) to the graphics processor (GPU). To solve that task, we utilized our in-house developed Fastvideo SDK, which works on NVIDIA GPU. This approach has significantly accelerated the necessary algorithms and greatly simplified the software architecture, because computations in this case no longer interfere with system functions based on the CPU. 
Let's look at the advantages of image processing on a GPU instead of a CPU:
Transferring computations to the graphics card does not mean that the CPU is completely free. The CPU is responsible for I/O and system control. The proposed solution is heterogeneous, since it uses all the available resources of both the CPU and GPU for image processing, which in turn leads to high performance. In addition to increasing the speed of image processing, using a graphics processor has allowed us to implement more complicated algorithms to increase the image quality and color reproduction. Our workflow is similar to that used in filmmaking, where the colors in the frame are given special attention. 
Fig.1. XIMEA xiB high performance machine vision cameras One of the best examples where this solution can be applied is image processing for XIMEA cameras. XIMEA manufactures high-quality industrial cameras with the latest image sensors, which provide exceptionally high data rates. The Fastvideo SDK solution offers an excellent approach for real-time image processing for high performance cameras.
Fig.2. Menzi Muck walking excavator XIMEA cameras are used, for example, in the Menzi Muck remote-controlled walking excavator. For this particular project, the Fastvideo SDK solution allowed:
Fig.3. Wind turbine inspection drone from Alerion with XIMEA camera Let's take as an example another project using XIMEA cameras: the wind turbine inspection drone from Alerion. This drone is intended to fully automate the inspection of wind turbines for damage. For this task, it is very important to ensure good quality of images, based on which a 3D model is subsequently built. Using XIMEA cameras in conjunction with the GPU image processing solution made it possible to achieve the required image quality and high processing speed, which in turn made it possible to automate the inspection process. As a result, the time spent on inspection of one turbine was reduced from 2-3 hours to 10 minutes. Here, of course, process automation played a big role. However, this would not have been possible without high processing speed and excellent image quality that allows even very small damage to be noticed. In conclusion, it’s worth noting the versatility of the Fastvideo SDK for GPU image processing: it can work on both stationary and mobile graphics cards. However, when choosing a solution for your task, don’t forget about the price-performance ratio. If you configure the solution to meet your needs (download source codes from GitHub), you’ll get high performance and high quality software for real time applications and avoid unnecessary costs for hardware. |