NeRF4SeRe: Neural Radiance Fields for Scene Reconstruction

Project of AI3604 Computer Vision, SJTU.
Wei Jiang * · Kangrui Mao * · Gu Zhang * · Haoyu Zhen *
\star = equal contribution

TODO

• installation guideline
• training and testing code for NeRF, INeRF and Instant-NGP
• implementation of NeRFusion
• NICE-SLAM integration
• visualizations

Installation

git clone --recursive https://github.com/NeRF-SeRe/SeRe.git
cd SeRe

1. Environment setup You should install google-sparsehash first following the instructions in torchsparse (e.g., sudo apt-get install libsparsehash-dev on Ubuntu and brew install google-sparsehash on Mac OS).

   Then install the dependencies:

   conda create -c conda-forge -n sere python=3.9
   pip install -r requirements.txt
   pip install git+https://github.com/facebookresearch/pytorch3d.git
   pip install --upgrade git+https://github.com/mit-han-lab/[email protected]
   export PYTHONPATH="/path/to/sere:$PYTHONPATH"

2. Dataset preparation Please refer to https://github.com/cvg/nice-slam to download the Replica Dataset. For SJTUers, you could download it from Replica-JBox.

   Then link the dataset

   ln -s /path/to/Replica data/replica

Training Pipeline

Adding --device x to specify the GPU device (default: 0).

1. Vanilla NeRF

   python run/main_nerf.py --config configs/nerf/replica/room0.txt

2. Instant-NGP

   python run/main_ngp.py data/replica/room0/ --workspace output/ngp/replica/room0/ --fp16 --cuda_ray

3. NeRFusion

   python run/main_nerfusion.py data/replica/room0 --fusion_path data/replica --workspace output/nerfusion/replica/room0 --cuda_ray --fp16 
   
   python run/main_nerfusion.py data/replica/room0 --fusion_path data/replica --fusion_scenes office0 office1 --workspace output/nerfusion/replica/room0 --cuda_ray --fp16 # change fusion_scenes to specify the scenes to be trained

4. Refine camera pose (INeRF)

   After you have trained a NeRF model, please put the checkpoints (its name should keep the same as model_name in config) under /ckpts/inerf. Then run:

   python run/main_inerf.py --config configs/inerf/replica/room0.txt

Evaluation

If you have already trained a NeRF model, just add --test to test it. eg.

python run/main_ngp.py data/replica/room0/ --workspace output/ngp/replica/room0/ --fp16 --cuda_ray --test # write images

python run/main_ngp.py data/replica/room0/ --workspace output/ngp/replica/room0/ --fp16 --cuda_ray --test --video # generate video

Acknowledgement

• nerf-pytorch
• inerf
• torch-ngp
• NeRFusion
• NeuralRecon

For TA

Our latest report and appendices are in docs.