3DGS-Calib: 3D Gaussian Splatting for Multimodal SpatioTemporal Calibration

IROS 2024 · Oral

Authors
Quentin Herau1,3Moussab Bennehar1Arthur Moreau2Nathan Piasco1Luis Roldão1Dzmitry Tsishkou1Cyrille Migniot4Pascal Vasseur5Cédric Demonceaux3

1 Noah’s Ark, Huawei Paris Research Center, France2 Noah’s Ark, Huawei London Research Center, United Kingdom3 ICB UMR CNRS 6303, Université de Bourgogne, France4 ImViA UR 7535, Université de Bourgogne, France5 MIS UR 4290, Université de Picardie Jules Verne, France

HuaweiICBImViAMISCNRSUniversité Bourgogne EuropeUniversité de Picardie Jules Verne

Anchor a neural Gaussian representation to LiDAR geometry, then refine camera extrinsics and synchronization through differentiable rendering.

Method

3DGS-Calib anchors Gaussian centers to accumulated LiDAR points. A neural network predicts Gaussian attributes; differentiable rasterization jointly optimizes that representation and the camera extrinsics and time offsets. Coarse-to-fine voxelization accelerates calibration.

3DGS-Calib architecture: LiDAR points anchor the Gaussians, and differentiable rendering provides the camera calibration signal.
3DGS-Calib architecture: LiDAR points anchor the Gaussians, and differentiable rendering provides the camera calibration signal.

Results from the paper

LiDAR–camera calibration on KITTI-360

The paper evaluates the front-left camera against LiDAR on three KITTI-360 sequences. This experiment uses 3,000 training steps and progressive voxel resolution. All methods use the paper’s evaluation setup.

Table IV · Spatial calibration, with temporal calibration disabled. Lower is better.
MethodRotation (°) ↓Translation (cm) ↓Training time (s) ↓
Pandey et al.11.6 ± 6.590.7 ± 29.5—
LCCNet1.86 ± 0.0989.5 ± 2.2—
MOISST4.92 ± 0.564.1 ± 6.81527
MOISST w/ cropping6.01 ± 1.275.0 ± 3.7860
3DGS-Calib0.45 ± 0.29.60 ± 2.1216

Qualitative calibration results

Pandey et al.

3DGS-Calib: Pandey et al. LiDAR-to-camera alignment

LCCNet

3DGS-Calib: LCCNet LiDAR-to-camera alignment

MOISST

3DGS-Calib: MOISST LiDAR-to-camera alignment

3DGS-Calib

3DGS-Calib: 3DGS-Calib LiDAR-to-camera alignment
LiDAR–camera alignment on KITTI-360, from the 3DGS-Calib paper. Paper ↗

Presentation & supplementary video

Research code

The open-source code release is coming soon. neural-calibration will provide MOISST, SOAC, 3DGS-Calib, MOISST++ and SOAC++ implementations, with raw-data setup, training and pose-export instructions. avoptim-benchmark will provide dataset processing and pose evaluation for KITTI-360, nuScenes, PandaSet and Waymo, supporting Original, refined and custom method poses.

The results shown here are reported paper results.

Citation

BibTeX
@misc{herau20243dgscalib,
      title={3DGS-Calib: 3D Gaussian Splatting for Multimodal SpatioTemporal Calibration},
      author={Quentin Herau and Moussab Bennehar and Arthur Moreau and Nathan Piasco and Luis Roldao and Dzmitry Tsishkou and Cyrille Migniot and Pascal Vasseur and Cédric Demonceaux},
      year={2024},
      eprint={2403.11577},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
Paper figure