Multi-modal image registration for robotic aerial inspection using mutual information

In this work, we present a novel two-stage template registration approach based on mutual information to precisely register multi-modal sensor data of photovoltaic modules as the first step in data processing for autonomous robotic inspection of renewable infrastructure. In stage 1 we construct a multiresolution transform net that samples the global warp parameters space. A branch-and-bound algorithm is used to efficiently search the transforms net until a candidate solution is determined that lies in a local-basin of attraction for the true optimum. The algorithm then switches to a robust and efficient trust-region descent method. The proposed search determines the warp that aligns the PV module template to the sensor image to subpixel accuracy with quadratic convergence. By using a mutual-information cost criterion, the same template can be used to register multiple sensing modalities such as thermal, near-infrared (NIR), hyperspectral, RGBD, etc., providing the operators with a suite of data that enables detailed diagnosis of faults and defects. The robustness of our approach is illustrated by performing mono-modal and multi-modal template registration for data collected from Australian solar farms. The experiments show good warp estimation as quantified by the intersection-of-union metric.