MOISST: Multimodal Optimization of Implicit Scene for SpatioTemporal Calibration

IROS 2023

Authors
Quentin Herau1,2Nathan Piasco1Moussab Bennehar1Luis Roldão1Dzmitry Tsishkou1Cyrille Migniot3Pascal Vasseur4Cédric Demonceaux2

1 Noah’s Ark, Huawei Paris Research Center, France2 ICB UMR CNRS 6303, Université de Bourgogne, France3 ImViA UR 7535, Université de Bourgogne, France4 MIS UR 4290, Université de Picardie Jules Verne, France

HuaweiICBImViAMISCNRSUniversité Bourgogne EuropeUniversité de Picardie Jules Verne
PaperPublisherPaper resultsCalibration code

Bring cameras and LiDAR into a shared spatial and temporal frame by optimizing a neural scene together with sensor calibration.

MOISST paper illustration: RGB and depth before and after calibration, with the sensor poses shown below.
MOISST paper illustration: RGB and depth before and after calibration, with the sensor poses shown below.

Method

MOISST uses a shared neural radiance field to connect camera images and LiDAR ranges. Differentiable rendering jointly optimizes the scene, rigid sensor extrinsics and clock offsets against a supplied continuous-time reference trajectory.

Published MOISST architecture. Camera and LiDAR observations connect through a common implicit scene.
Published MOISST architecture. Camera and LiDAR observations connect through a common implicit scene.

Results from the paper

Spatial calibration on KITTI-360

The paper evaluates sequences 0, 1, 2 and 4 with the front-left camera as reference. This spatial-only experiment starts with translation and rotation perturbations; the reported errors average the last ten epochs.

Table I · Front-right camera and LiDAR. Values retain the paper’s reported ± terms.
SequenceCamera translation (cm) ↓Camera rotation (°) ↓LiDAR translation (cm) ↓LiDAR rotation (°) ↓
02.3 ± 0.60.09 ± 0.028.2 ± 1.80.21 ± 0.08
11.6 ± 0.40.09 ± 0.027.3 ± 2.10.41 ± 0.08
21.2 ± 0.60.04 ± 0.0313.5 ± 1.80.18 ± 0.10
42.2 ± 0.70.08 ± 0.069.9 ± 2.60.30 ± 0.10

Source: original paper table ↗

Presentation

Research code

neural-calibration prepares raw KITTI-360, nuScenes, PandaSet and Waymo data for independent MOISST, SOAC, 3DGS-Calib, MOISST++ and SOAC++ implementations, with training and sensor-pose export instructions. avoptim-benchmark prepares the twenty benchmark sequences and evaluates Original, provided MOISST++ or custom method poses across ten rendering, structure-from-motion and geometry metrics. Original poses and observation metadata are generated locally from authorized dataset downloads; the provided optimized bundle contains pose matrices only. Training and benchmark preparation use separate commands documented in the repositories.

The results shown here are reported paper results. See the repositories for implementation details and validation scope.

Citation

BibTeX
@inproceedings{herau2023moisst,
  title={MOISST: Multimodal Optimization of Implicit Scene for SpatioTemporal Calibration},
  author={Herau, Quentin and Piasco, Nathan and Bennehar, Moussab and Rold{\~a}o, Luis and Tsishkou, Dzmitry and Migniot, Cyrille and Vasseur, Pascal and Demonceaux, C{\'e}dric},
  booktitle={2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  pages={1810--1817},
  year={2023},
  organization={IEEE}
}

Download BibTeX

Paper figure