A target-free approach to estimate thermal camera pose in LiDAR scenes of feature-deficient environments

Abstract

Modern animal behaviourists are able to collect vast amounts of data from multiple sensors. Sensor fusion however still remains a challenge given the unique sensor types biologists use. Here we study thermal camera-LiDAR alignment. While LiDAR-RGB scene alignment is well-established in feature-rich scenes with reliable inter-camera correspondences, thermal scenes are often captured with low-resolution devices and suffer from halo and history effects, and thus require sensor-specific algorithms. To deal with this, we present the depth-map correspondence (DMCP) algorithm. DMCP is a semi-automatic algorithm to fuse Li-DAR with feature-deficient thermal scenes without explicit calibration objects. The user annotates at least 4 corresponding points on the thermal image and a depth map of the LiDAR scene, which allows us to compute the position and orientation of the thermal camera within the scene. We quantify the accuracy of alignment using 2D reprojection error and 3D nearest-neighbour distances of known world points across experimental trials. A median reprojection error of 7.7 (95%ile: 3.7-23.5) pixels was achieved. Known objects lying on the LiDAR mesh showed a median nearest-neighbour mesh distance of 0.12 m (95%ile: 0.007-5.61 m). We present the first such alignment of feature-deficient thermal scenes and LiDAR data. DMCP has two advantages 1) it is computationally straightforward to implement and 2) requires minimal user annotation to work. DMCP could be broadly applied to any type of mesh-image alignment problem. Finally we provide DMCP and the Ushichka dataset as a baseline for further research on the challenging problem of subterranean thermal-LiDAR alignment.