Software for Hamamatsu ORCA Processing on GPU

Author: Fyodor Serzhenko

Scientific research demands modern cameras with low noise, high resolution, frame rate and bit depth. Such imaging solutions are indispensable in microscopy, experiments with cold atom gases, astronomy, photonics, etc. Apart from outstanding hardware there is a need for high performance software to process streams in realtime with high precision.

Hamamatsu Photonincs company is a world leader in scientific cameras, light sources, photo diodes and advanced imaging applications. For high performance scientific cameras and advanced imaging applications, Hamamatsu introduced ORCA cameras with outstanding features. ORCA cameras are high precision instruments for scientific imaging due to on-board FPGA processing enabling intelligent data reduction, pixel-level calibrations, increased USB 3.0 frame rates, purposeful and innovative triggering capabilities, patented lightsheet read out modes and individual camera noise characterization.

ORCA-Flash4.0 cameras have always provided the advantage of low camera noise. In quantitative applications, like single molecule imaging and super resolution microscopy imaging, fully understanding camera noise is also important. Every ORCA-Flash4.0 V3 is carefully calibrated to deliver outstanding linearity, especially at low light, to offer improved photo response non-uniformity (PRNU) and dark signal non-uniformity (DSNU), to minimize pixel differences and to reduce fixed pattern noise (FPN).

The ORCA-Flash4.0 V3 includes patented Lightsheet Readout Mode, which takes advantage of sCMOS rolling shutter readout to enhance the quality of lightsheet images. When paired with W-VIEW GEMINI image splitting optics, a single ORCA-Flash4.0 V3 camera becomes a powerful dual wavelength imaging device. In "W-VIEW Mode" each half of the image sensor can be exposed independently, facilitating balanced dual color imaging with a single camera. And this feature can be combined with the new and patented "Dual Lightsheet Mode" to offer simultaneous dual wavelength lightsheet microscopy.

 

Hamamatsu software for ORCA cameras

 

Applications for Hamamatsu ORCA cameras

There are quite a lot of scientific imaging tasks which could be solved with Hamamatsu ORCA cameras:

  • Digital Microscopy
  • Light Sheet Fluorescence Microscopy
  • Live-Cell Microscopy and Live-Cell Imaging
  • Laser Scanning Confocal Microscopy
  • Biophysics and Biophotonics
  • Biological and Biomedical Sciences
  • Bioimaging and Biosensing
  • Neuroimaging

 

Hamamatsu ORCA-Flash4.0 V3 Digital CMOS camera

Hamamatsu ORCA-Flash4.0 V3 Digital CMOS camera (image from https://camera.hamamatsu.com/jp/en/product/search/C13440-20CU/index.html)

Hamamatsu ORCA-Flash4.0 V3 Digital CMOS camera: C13440-20CU

Part Number C13440-20CU
Resolution 4 MPix, 2048 × 2048 pixels
Pixel size 6.5 μm × 6.5 μm
Effective area 13.312 mm × 13.312 mm
Sensor type sCMOS
Quantum efficiency 82% (peak QE: @560 nm)
Full well capacity 30 000 electrons (typ.)
Readout speed 100 frames/s (Full resolution, standard scan, Camera Link)
40 frames/s (Full resolution, standard scan, USB 3.0, 16-bit)
53 frames/s (Full resolution, standard scan, USB 3.0, 12-bit)
80 frames/s (Full resolution, standard scan, USB 3.0, 8-bit)
Readout noise Standard scan (at 100 frames/s, typ.):1.6 electrons rms (1.0 electrons median)
Slow scan (at 30 frames/s, typ.): 1.4 electrons rms (0.8 electrons median)
Cooling method Peltier cooling
Cooling temperature Forced air (Ambient at +20 ℃): -10 ℃
Water (+20 ℃): -10 ℃
Water (+15 ℃): -30 ℃
Dark current 0.06 electrons/pixel/s (Air Cooled to -10° C) (typ.)
0.06 electrons/pixel/s (Water Cooled to -10° C) (typ.)
0.006 electrons/pixel/s (Water Cooled to -30° C) (typ.)
Dynamic range 37 000:1 (typ.)
Interface Camera Link / USB 3.0
A/D converter 16-bit / 12-bit / 8-bit
Lens mount C-mount
Readout Modes Normal Area, Lightsheet, W-VIEW Mode, Dual Lightsheet
Binning 2×2, 4×4
Hot Pixel Correction Off, Low, Medium, High
Dark Signal Non-Uniformity (DSNU) 0.3 electrons rms

Image processing for Hamamatsu ORCA-Flash4.0 V3 Digital CMOS camera

That camera generates quite high data rate. Maximum performance for Hamamatsu ORCA-Flash4.0 V3 could be evaluated as 100 fps * 4 MPix * 2 Byte/Pix = 800 MByte/s. As soon as these are 16-bit monochrome frames, that high data rate could be a bottleneck to save such streams to SSD for two-camera system for long-term recording, which is quite usual for microscopy applications.

If we consider one-day recoding duration, storage for such a stream could be a problem. That two-camera system generates 5.76 TB data per hour and it could be a good idea to implement realtime compression to cut storage cost. To compress 16-bit frames, we can't utilize either JPEG or H.265 encoding algorithms because they don't support more than 12-bit data. The best choice here is JPEG2000 compression algorithm which is working natively with 16-bit images. On NVIDIA GeForce GTX 1080 we've got the performance around 240 fps for lossy JPEG2000 encoding with compression ratio around 20. This is the result that we can't achieve on CPU because corresponding JPEG2000 implementations (OpenJPEG, Jasper, J2K, Kakadu) are much slower. Here you can see JPEG2000 benchmark comparison for widespread J2K encoders.

JPEG2000 lossless compression algorithm is also available, but it offers much less compression ratio, usually in the range of 2-2.5 times. Still, it's useful option to store original compressed data without any losses which could be mandatory for particular image processing workflow. In any way, lossless compression makes data rate less, so it's always good for storage and performance issues.

Optimal compression ratio for lossy JPEG2000 encoding should be defined by checking different quality metrics and their correspondence to a particular task to be solved. Still, there is no good alternative for fast JPEG2000 compression for 16-bit data, so JPEG2000 looks as the best fit. We would also recommend to add the following image processing modules to the full pipeline to get better image quality:

  • Dynamic Bad Pixel Correction
  • Data linearization with 1D LUT
  • Dark Frame Subtraction
  • Flat Field Correction (vignette removal)
  • White/Black Points
  • Exposure Correction
  • Curves and Levels
  • Denoising
  • Crop, Flip/Flop, Rotate 90/180/270, Resize
  • Geometric transforms, Rotation to an arbitrary angle
  • Sharp
  • Gamma Correction
  • Realtime Histogram and Parade
  • Mapping and monitor output
  • Output JPEG2000 encoding (lossless or lossy)

The above image processing pipeline could be fully implemented on GPU to achieve realtime performance or even faster. It could be done with Fastvideo SDK and NVIDIA GPU. That SDK is supplied with sample applications in source codes, so user can create his own GPU-based application very fast. Fastvideo SDK is avalilable for Windows, Linux, L4T.

There is also a gpu-camera-sample application which is based on Fastvideo SDK. You can download source codes and/or binaries for Windows from the following link on Github - gpu camera sample. Binaries are able to work with raw images in PGM format (8/12/16-bit), even without a camera. User can add support for Hamamatsu cameras to process images in realtime on NVIDIA GPU.

orca image processing sdk

Fastvideo SDK to process on GPU raw images from Hamamatsu ORCA sCMOS cameras

The performance of JPEG2000 codec strongly depends on GPU, image content, encoding parameters and complexity of the full image processing pipeline. To scale the performance, user can also utilize several GPUs for image processing at the same time. Multiple GPU processing option is the part of Fastvideo SDK.

If you have any questions, please fill the form below with your task description and send us your sample images for evaluation.

Links

Hamamatsu ORCA-Flash4.0 V3 Digital sCMOS camera

GPU Software for camera applications

JPEG2000 Codec on NVIDIA GPU

Image and Video Processing SDK for NVIDIA GPUs

Fastvideo SDK vs NVIDIA NPP Library

GPU Software for machine vision and industrial cameras

Contact Form

This form collects your name and email. Check out our Privacy Policy on how we protect and manage your personal data.