nbv_reconstruction/Readme.md

# Next Best View for Reconstruction

## 1. Setup Environment
### 1.1 Install Main Project
```bash
mkdir nbv_rec
cd nbv_rec
git clone https://git.hofee.top/hofee/nbv_reconstruction.git
```
### 1.2 Install PytorchBoot
the environment is based on PytorchBoot, clone and install it from [PytorchBoot](https://git.hofee.top/hofee/PyTorchBoot.git)
```bash
git clone https://git.hofee.top/hofee/PyTorchBoot.git
cd PyTorchBoot
pip install .
cd ..
```
### 1.3 Install Blender (Optional)
If you want to render your own dataset as described in [section 2. Render Datasets](#2-render-datasets), you'll need to install Blender version 4.0 from [Blender Release](https://download.blender.org/release/Blender4.0/). Here is an example of installing Blender on Ubuntu:
```bash
wget https://download.blender.org/release/Blender4.0/blender-4.0.2-linux-x64.tar.xz
tar -xvf blender-4.0.2-linux-x64.tar.xz
```
If blender is not in your PATH, you can add it by:
```bash
export PATH=$PATH:/path/to/blender/blender-4.0.2-linux-x64
```
To run the blender script, you need to install the `pyyaml` and `scipy` package into your blender python environment. Run the following command to print the python path of your blender:
```bash
./blender -b --python-expr "import sys; print(sys.executable)"
```
Then copy the python path `/path/to/blender_python` shown in the output and run the following command to install the packages:
```bash
/path/to/blender_python -m pip install pyyaml scipy
```
### 1.4 Install Blender Render Script (Optional)
Clone the script from [nbv_rec_blender_render](https://git.hofee.top/hofee/nbv_rec_blender_render.git) and rename it to `blender`:
```bash
git clone https://git.hofee.top/hofee/nbv_rec_blender_render.git
mv nbv_rec_blender_render blender
```

### 1.5 Check Dependencies
Switch to the project root directory and run `pytorch-boot scan` or `ptb scan` to check if all dependencies are installed:
```bash
cd nbv_reconstruction
pytorch-boot scan
# or
ptb scan
```
If you see project structure information in the output, it means all dependencies are correctly installed. Otherwise, you may need to run `pip install xxx` to install the missing packages.

## 2. Render Datasets (Optional)
### 2.1 Download Object Mesh Models
Download the mesh models divided into three parts from: 
 - [object_meshes_part1.zip](None)
 - [object_meshes_part2.zip](https://pan.baidu.com/s/1pBPhrFtBwEGp1g4vwsLIxA?pwd=1234)
 - [object_meshes_part3.zip](https://pan.baidu.com/s/1peE8HqFFL0qNFhM5OC69gA?pwd=1234)

or download the whole dataset from [object_meshes.zip](https://pan.baidu.com/s/1ilWWgzg_l7_pPBv64eSgzA?pwd=1234)

Download the table model from [table.obj](https://pan.baidu.com/s/1sjjiID25Es_kmcdUIjU_Dw?pwd=1234)

### 2.2 Set Render Configurations
Open file `configs/local/view_generate_config.yaml` and modify the parameters to fit your needs. You are required to at least set the following parameters in `runner-generate`:
 - `object_dir`: the directory of the downloaded object mesh models
 - `output_dir`: the directory to save the rendered dataset
 - `table_model_path`: the path of the downloaded table model

### 2.3 Render Dataset

There are two ways to render the dataset:

#### 2.3.1 Render with Visual Monitoring

If you want to visually monitor the rendering progress and machine resource usage:

1. In the terminal, run:
   ```
   ptb ui
   ```
2. Open your browser and visit http://localhost:5000
3. Navigate to `Project Dashboard - Project Structure - Applications - generate_view`
4. Click the `Run` button to execute the rendering script

#### 2.3.2 Render in Terminal

If you don't need visual monitoring and prefer to run the rendering process directly in the terminal, simply run:

```
ptb run generate_view
```

This command will start the rendering process without launching the UI.

## 3. Preprocess

⚠️ The preprocessing code is currently not managed by `PytorchBoot`. To run the preprocessing:

1. Open the `./preprocess/preprocessor.py` file.
2. Locate the `if __name__ == "__main__":` block at the bottom of the file.
3. Specify the dataset folder by setting `root = "path/to/your/dataset"`.
4. Run the preprocessing script directly:

   ```
   python ./preprocess/preprocessor.py
   ```

This will preprocess the data in the specified dataset folder.

## 4. Generate Strategy Label

### 4.1 Set Configuration

Open the file `configs/local/strategy_generate_config.yaml` and modify the parameters to fit your needs. You are required to at least set the following parameter:

- `datasets.OmniObject3d.root_dir`: the directory of your dataset

### 4.2 Generate Strategy Label

There are two ways to generate the strategy label:

#### 4.2.1 Generate with Visual Monitoring

If you want to visually monitor the generation progress and machine resource usage:

1. In the terminal, run:
   ```
   ptb ui
   ```
2. Open your browser and visit http://localhost:5000
3. Navigate to Project Dashboard - Project Structure - Applications - generate_strategy
4. Click the `Run` button to execute the generation script

#### 4.2.2 Generate in Terminal

If you don't need visual monitoring and prefer to run the generation process directly in the terminal, simply run:

```
ptb run generate_strategy
```

This command will start the strategy label generation process without launching the UI.

## 5. Train

### 5.1 Set Configuration

Open the file `configs/local/train_config.yaml` and modify the parameters to fit your needs. You are required to at least set the following parameters in the `experiment` section:

```yaml
experiment:
  name: your_experiment_name
  root_dir: path/to/your/experiment_dir
  use_checkpoint: False # if True, the checkpoint will be loaded
  epoch: 600 # specific epoch to load, -1 stands for last epoch
  max_epochs: 5000 # maximum epochs to train
  save_checkpoint_interval: 1 # save checkpoint interval
  test_first: True # if True, test process will be performed before training at each epoch
```

Adjust these parameters according to your training requirements.


### 5.2 Start Training

There are two ways to start the training process:

#### 5.2.1 Train with Visual Monitoring

If you want to visually monitor the training progress and machine resource usage:

1. In the terminal, run:
   ```
   ptb ui
   ```
2. Open your browser and visit http://localhost:5000
3. Navigate to Project Dashboard - Project Structure - Applications - train
4. Click the `Run` button to start the training process

#### 5.2.2 Train in Terminal

If you don't need visual monitoring and prefer to run the training process directly in the terminal, simply run:

```
ptb run train
```

This command will start the training process without launching the UI.

## 6. Evaluation
...