SOAC: Spatio-Temporal Overlap-Aware Multi-Sensor Calibration using Neural Radiance Fields

CVPR 2024

Authors
Quentin Herau1,2Nathan Piasco1Moussab Bennehar1Luis Roldão1Dzmitry Tsishkou1Cyrille Migniot3Pascal Vasseur4Cédric Demonceaux2

1 Noah’s Ark, Huawei Paris Research Center, France2 ICB UMR CNRS 6303, Université de Bourgogne, France3 ImViA UR 7535, Université de Bourgogne, France4 MIS UR 4290, Université de Picardie Jules Verne, France

HuaweiICBImViAMISCNRSUniversité Bourgogne EuropeUniversité de Picardie Jules Verne
PaperCVPR proceedingsPaper resultsCode (coming soon)

Calibrate where sensors see the same scene. Per-camera neural fields and overlap-aware registration connect multiple cameras and LiDAR.

Method

SOAC trains camera-specific neural fields and alternates scene fitting with sensor registration. Visibility grids restrict cross-sensor calibration to overlapping scene regions, allowing camera and LiDAR observations to contribute where they share support.

Published SOAC architecture: camera-specific fields and overlap-aware cross-sensor registration.
Published SOAC architecture: camera-specific fields and overlap-aware cross-sensor registration.

Results from the paper

LiDAR–camera calibration

The paper compares the front-left camera and LiDAR on KITTI-360, and the front camera and 360° LiDAR on PandaSet. The initial errors are 50 cm translation and 5° rotation on each axis.

Table 2 · Spatial calibration, without temporal error. Lower is better.
MethodKITTI-360 rotation (°) ↓KITTI-360 translation (cm) ↓PandaSet rotation (°) ↓PandaSet translation (cm) ↓
Pandey et al.11.8 ± 5.4143 ± 10915.4 ± 0.8139 ± 17.5
LCCNet1.9 ± 0.195.8 ± 7.714.3 ± 3.4370 ± 11.6
MOISST0.2 ± 0.110.0 ± 9.82.8 ± 2.356.4 ± 17.2
SOAC0.3 ± 0.27.8 ± 3.51.3 ± 0.829.4 ± 13.6

Source: original paper table ↗

Qualitative calibration results

Dataset reference (GT)

SOAC: Dataset reference (GT) LiDAR-to-camera alignmentSOAC: Dataset reference (GT) LiDAR-to-camera alignment detail

LCCNet

SOAC: LCCNet LiDAR-to-camera alignmentSOAC: LCCNet LiDAR-to-camera alignment detail

SOAC

SOAC: SOAC LiDAR-to-camera alignmentSOAC: SOAC LiDAR-to-camera alignment detail
LiDAR–camera alignment on KITTI-360, from the SOAC paper. Paper ↗

Presentation

Research code

The open-source code release is coming soon. neural-calibration will provide MOISST, SOAC, 3DGS-Calib, MOISST++ and SOAC++ implementations, with raw-data setup, training and pose-export instructions. avoptim-benchmark will provide dataset processing and pose evaluation for KITTI-360, nuScenes, PandaSet and Waymo, supporting Original, refined and custom method poses.

The results shown here are reported paper results.

Citation

BibTeX
@inproceedings{herau2024soac,
  author = {Herau, Quentin and Piasco, Nathan and Bennehar, Moussab and Rold{\~a}o, Luis and Tsishkou, Dzmitry and Migniot, Cyrille and Vasseur, Pascal and Demonceaux, C{\'e}dric},
  title = {{SOAC}: Spatio-Temporal Overlap-Aware Multi-Sensor Calibration using Neural Radiance Fields},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2024},
  month = {June},
  pages = {15131--15140}
}

Download BibTeX

Paper figure