Spin-UP: Spin Light for Natural Light Uncalibrated Photometric Stereo

Zongrui Li1, *, Zhan Lu1, 2, *, Haojie Yan2, Boxin Shi3, Gang Pan2, Qian Zheng2, Xudong Jiang1,
1Nanyang Technological University, 2Zhejiang University, 3Peking University
CVPR 2024

*Equal Contribution

Abstract

Natural Light Uncalibrated Photometric Stereo (NaUPS) relieves the strict environment and light assumptions in classical Uncalibrated Photometric Stereo (UPS) methods. However, due to the intrinsic ill-posedness and high-dimensional ambiguities, addressing NaUPS is still an open question. Existing works impose strong assumptions on the environment lights and objects' material, restricting the effectiveness in more general scenarios. Alternatively, some methods leverage supervised learning with intricate models while lacking interpretability, resulting in a biased estimation. In this work, we proposed Spin Light Uncalibrated Photometric Stereo (Spin-UP), an unsupervised method to tackle NaUPS in various environment lights and objects. The proposed method uses a novel setup that captures the object's images on a rotatable platform, which mitigates NaUPS's ill-posedness by reducing unknowns and provides reliable priors to alleviate NaUPS's ambiguities. Leveraging neural inverse rendering and the proposed training strategies, Spin-UP recovers surface normals, environment light, and isotropic reflectance under complex natural light with low computational cost. Experiments have shown that Spin-UP outperforms other supervised / unsupervised NaUPS methods and achieves state-of-the-art performance on synthetic and real-world datasets.

Overview

The proposed Spin-UP handles natural light photometric stereo based on a new setup. We highlight our contribution as:

  1. We design a novel setup for NaUPS, which reduces unknowns of light representation and facilitates solving NaUPS in an unsupervised manner;
  2. We introduce a light prior, which leverages an object’s occluding boundaries to initialize a reliable environment light. Based on the setup and light prior, we propose the unsupervised NaUPS method named Spin-UP;
  3. We present two training strategies for fast training and convergence of Spin-UP;
The proposed Spin-UP produces highly accurate surface normal given images of object captured under natural light in an unsupervised manner, even comparable to the SOTA supervised method.

Teaser



Spin Light Setup

We record a video for an object under an environment light by rotating the object together with a linear, perspective camera in 360° on a rotatable platform. The relative positions and orientations between camera and object remain fixed during rotation.

We then extract around 50 images from the recorded video, regarding as our training data.

Teaser



Light Initialization Method

Based on the proposed setup, we derive a rough light map given the observed images.




Training Strategies

We proposed two training strategies (top: interval sampling, bottom: shrinking range computing) to improve the training efficiency and alleviate aliasing issues.

Teaser



Video Presentation




BibTeX

@inproceedings{li2024spinup,
      title={Spin-UP: Spin Light for Natural Light Uncalibrated Photometric Stereo},
      author={Li, Zongrui and Lu, Zhan and Yan, Haojie and Shi, Boxin and Pan, Gang and Zheng, Qian and Jiang, Xudong},
      booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
      year={2024}}