diff --git a/Readme.md b/Readme.md new file mode 100644 index 0000000..f916b5a --- /dev/null +++ b/Readme.md @@ -0,0 +1,192 @@ +# Next Best View for Reconstruction + +## 1. Setup Environment +### 1.1 Install Main Project +```bash +mkdir nbv_rec +cd nbv_rec +git clone https://git.hofee.top/hofee/nbv_reconstruction.git +``` +### 1.2 Install PytorchBoot +the environment is based on PytorchBoot, clone and install it from [PytorchBoot](https://git.hofee.top/hofee/PyTorchBoot.git) +```bash +git clone https://git.hofee.top/hofee/PyTorchBoot.git +cd PyTorchBoot +pip install . +cd .. +``` +### 1.3 Install Blender (Optional) +If you want to render your own dataset as described in [section 2. Render Datasets](#2-render-datasets), you'll need to install Blender version 4.0 from [Blender Release](https://download.blender.org/release/Blender4.0/). Here is an example of installing Blender on Ubuntu: +```bash +wget https://download.blender.org/release/Blender4.0/blender-4.0.2-linux-x64.tar.xz +tar -xvf blender-4.0.2-linux-x64.tar.xz +``` +If blender is not in your PATH, you can add it by: +```bash +export PATH=$PATH:/path/to/blender/blender-4.0.2-linux-x64 +``` +To run the blender script, you need to install the `pyyaml` and `scipy` package into your blender python environment. Run the following command to print the python path of your blender: +```bash +./blender -b --python-expr "import sys; print(sys.executable)" +``` +Then copy the python path `/path/to/blender_python` shown in the output and run the following command to install the packages: +```bash +/path/to/blender_python -m pip install pyyaml scipy +``` +### 1.4 Install Blender Render Script (Optional) +Clone the script from [nbv_rec_blender_render](https://git.hofee.top/hofee/nbv_rec_blender_render.git) and rename it to `blender`: +```bash +git clone https://git.hofee.top/hofee/nbv_rec_blender_render.git +mv nbv_rec_blender_render blender +``` + +### 1.5 Check Dependencies +Switch to the project root directory and run `pytorch-boot scan` or `ptb scan` to check if all dependencies are installed: +```bash +cd nbv_reconstruction +pytorch-boot scan +# or +ptb scan +``` +If you see project structure information in the output, it means all dependencies are correctly installed. Otherwise, you may need to run `pip install xxx` to install the missing packages. + +## 2. Render Datasets (Optional) +### 2.1 Download Object Mesh Models +Download the mesh models divided into three parts from: + - [object_meshes_part1.zip](None) + - [object_meshes_part2.zip](https://pan.baidu.com/s/1pBPhrFtBwEGp1g4vwsLIxA?pwd=1234) + - [object_meshes_part3.zip](https://pan.baidu.com/s/1peE8HqFFL0qNFhM5OC69gA?pwd=1234) + +or download the whole dataset from [object_meshes.zip](https://pan.baidu.com/s/1ilWWgzg_l7_pPBv64eSgzA?pwd=1234) + +Download the table model from [table.obj](https://pan.baidu.com/s/1sjjiID25Es_kmcdUIjU_Dw?pwd=1234) + +### 2.2 Set Render Configurations +Open file `configs/local/view_generate_config.yaml` and modify the parameters to fit your needs. You are required to at least set the following parameters in `runner-generate`: + - `object_dir`: the directory of the downloaded object mesh models + - `output_dir`: the directory to save the rendered dataset + - `table_model_path`: the path of the downloaded table model + +### 2.3 Render Dataset + +There are two ways to render the dataset: + +#### 2.3.1 Render with Visual Monitoring + +If you want to visually monitor the rendering progress and machine resource usage: + +1. In the `view_generate_config.yaml` file, under the `runner-generate` section, run: + ``` + ptb ui + ``` +2. Open your browser and visit http://localhost:5000 +3. Navigate to `Project Dashboard - Project Structure - Applications - generate_view` +4. Click the `Run` button to execute the rendering script + +#### 2.3.2 Render in Terminal + +If you don't need visual monitoring and prefer to run the rendering process directly in the terminal, simply run: + +``` +ptb run generate_view +``` + +This command will start the rendering process without launching the UI. + +## 3. Preprocess + +⚠️ The preprocessing code is currently not managed by `PytorchBoot`. To run the preprocessing: + +1. Open the `./preprocess/preprocessor.py` file. +2. Locate the `if __name__ == "__main__":` block at the bottom of the file. +3. Specify the dataset folder by setting `root = "path/to/your/dataset"`. +4. Run the preprocessing script directly: + + ``` + python ./preprocess/preprocessor.py + ``` + +This will preprocess the data in the specified dataset folder. + +## 4. Generate Strategy Label + +### 4.1 Set Configuration + +Open the file `configs/local/strategy_generate_config.yaml` and modify the parameters to fit your needs. You are required to at least set the following parameter: + +- `datasets.OmniObject3d.root_dir`: the directory of your dataset + +### 4.2 Generate Strategy Label + +There are two ways to generate the strategy label: + +#### 4.2.1 Generate with Visual Monitoring + +If you want to visually monitor the generation progress and machine resource usage: + +1. In the terminal, run: + ``` + ptb ui + ``` +2. Open your browser and visit http://localhost:5000 +3. Navigate to Project Dashboard - Project Structure - Applications - generate_strategy +4. Click the `Run` button to execute the generation script + +#### 4.2.2 Generate in Terminal + +If you don't need visual monitoring and prefer to run the generation process directly in the terminal, simply run: + +``` +ptb run generate_strategy +``` + +This command will start the strategy label generation process without launching the UI. + +## 5. Train + +### 5.1 Set Configuration + +Open the file `configs/local/train_config.yaml` and modify the parameters to fit your needs. You are required to at least set the following parameters in the `experiment` section: + +```yaml +experiment: + name: your_experiment_name + root_dir: path/to/your/experiment_dir + use_checkpoint: False # if True, the checkpoint will be loaded + epoch: 600 # specific epoch to load, -1 stands for last epoch + max_epochs: 5000 # maximum epochs to train + save_checkpoint_interval: 1 # save checkpoint interval + test_first: True # if True, test process will be performed before training at each epoch +``` + +Adjust these parameters according to your training requirements. + + +### 5.2 Start Training + +There are two ways to start the training process: + +#### 5.2.1 Train with Visual Monitoring + +If you want to visually monitor the training progress and machine resource usage: + +1. In the terminal, run: + ``` + ptb ui + ``` +2. Open your browser and visit http://localhost:5000 +3. Navigate to Project Dashboard - Project Structure - Applications - train +4. Click the `Run` button to start the training process + +#### 5.2.2 Train in Terminal + +If you don't need visual monitoring and prefer to run the training process directly in the terminal, simply run: + +``` +ptb run train +``` + +This command will start the training process without launching the UI. + +## 6. Evaluation +... diff --git a/TODO.md b/TODO.md deleted file mode 100644 index bc87a3d..0000000 --- a/TODO.md +++ /dev/null @@ -1,22 +0,0 @@ -# TODO -## 预处理数据 -### 1. 生成view阶段 -**input**: 物体mesh - -### 2. 生成label阶段 -**input**: 目标物体点云、目标物体点云法线、桌面扫描点、被拍到的桌面扫描点 - -**可以删掉的数据**: mask、normal - -### 3. 训练阶段 -**input**: 完整点云、pose、label - -**可以删掉的数据**:depth - -### view生成后 -预处理目标物体点云、目标物体点云法线、桌面扫描点、被拍到的桌面扫描点、完整点云 - -删除depth、mask、normal - -### label生成后 -只上传:完整点云、pose、label \ No newline at end of file diff --git a/configs/local/strategy_generate_config.yaml b/configs/local/strategy_generate_config.yaml index 7b88d9c..4a2a0e7 100644 --- a/configs/local/strategy_generate_config.yaml +++ b/configs/local/strategy_generate_config.yaml @@ -14,12 +14,6 @@ runner: voxel_threshold: 0.003 soft_overlap_threshold: 0.3 hard_overlap_threshold: 0.6 - filter_degree: 75 - to_specified_dir: True # if True, output_dir is used, otherwise, root_dir is used - save_points: True - load_points: True - save_best_combined_points: False - save_mesh: True overwrite: False seq_num: 15 dataset_list: @@ -27,11 +21,8 @@ runner: datasets: OmniObject3d: - #"/media/hofee/data/data/temp_output" root_dir: /media/hofee/repository/full_data_output - model_dir: /media/hofee/data/data/scaled_object_meshes from: 0 to: -1 # -1 means end - #output_dir: "/media/hofee/data/data/label_output" diff --git a/configs/local/view_generate_config.yaml b/configs/local/view_generate_config.yaml index e2f8779..2f67565 100644 --- a/configs/local/view_generate_config.yaml +++ b/configs/local/view_generate_config.yaml @@ -7,12 +7,12 @@ runner: name: debug root_dir: experiments generate: - port: 5004 + port: 5000 from: 0 - to: 1 # -1 means all - object_dir: H:\\AI\\Datasets\\scaled_object_box_meshes + to: -1 # -1 means all + object_dir: H:\\AI\\Datasets\\object_meshes_part2 table_model_path: "H:\\AI\\Datasets\\table.obj" - output_dir: C:\\Document\\Local Project\\nbv_rec\\nbv_reconstruction\\temp + output_dir: C:\\Document\\Datasets\\nbv_rec_part2 binocular_vision: true plane_size: 10 max_views: 512 diff --git a/configs/server/server_split_dataset_config.yaml b/configs/server/server_split_dataset_config.yaml deleted file mode 100644 index d3ddb5d..0000000 --- a/configs/server/server_split_dataset_config.yaml +++ /dev/null @@ -1,22 +0,0 @@ - -runner: - general: - seed: 0 - device: cpu - cuda_visible_devices: "0,1,2,3,4,5,6,7" - - experiment: - name: debug - root_dir: "experiments" - - split: # - root_dir: "/home/data/hofee/project/nbv_rec/data/nbv_rec_data_512_preproc_npy" - type: "unseen_instance" # "unseen_category" - datasets: - OmniObject3d_train: - path: "../data/sample_for_training_preprocessed/OmniObject3d_train.txt" - ratio: 0.9 - - OmniObject3d_test: - path: "../data/sample_for_training_preprocessed/OmniObject3d_test.txt" - ratio: 0.1 \ No newline at end of file diff --git a/configs/server/server_strategy_generate_config.yaml b/configs/server/server_strategy_generate_config.yaml deleted file mode 100644 index ad8e161..0000000 --- a/configs/server/server_strategy_generate_config.yaml +++ /dev/null @@ -1,37 +0,0 @@ - -runner: - general: - seed: 0 - device: cpu - cuda_visible_devices: "0,1,2,3,4,5,6,7" - - - experiment: - name: debug - root_dir: "experiments" - - generate: - voxel_threshold: 0.003 - soft_overlap_threshold: 0.3 - hard_overlap_threshold: 0.6 - filter_degree: 75 - to_specified_dir: True # if True, output_dir is used, otherwise, root_dir is used - save_points: True - load_points: True - save_best_combined_points: False - save_mesh: True - overwrite: False - seq_num: 15 - dataset_list: - - OmniObject3d - -datasets: - OmniObject3d: - #"/media/hofee/data/data/temp_output" - root_dir: /data/hofee/data/packed_preprocessed_data - model_dir: /media/hofee/data/data/scaled_object_meshes - from: 0 - to: -1 # -1 means end - #output_dir: "/media/hofee/data/data/label_output" - - diff --git a/runners/strategy_generator.py b/runners/strategy_generator.py index ca40f99..cd17860 100644 --- a/runners/strategy_generator.py +++ b/runners/strategy_generator.py @@ -22,13 +22,7 @@ class StrategyGenerator(Runner): "app_name": "generate_strategy", "runner_name": "strategy_generator" } - self.to_specified_dir = ConfigManager.get("runner", "generate", "to_specified_dir") - self.save_best_combined_pts = ConfigManager.get("runner", "generate", "save_best_combined_points") - self.save_mesh = ConfigManager.get("runner", "generate", "save_mesh") - self.load_pts = ConfigManager.get("runner", "generate", "load_points") - self.filter_degree = ConfigManager.get("runner", "generate", "filter_degree") self.overwrite = ConfigManager.get("runner", "generate", "overwrite") - self.save_pts = ConfigManager.get("runner","generate","save_points") self.seq_num = ConfigManager.get("runner","generate","seq_num") diff --git a/utils/data_load.py b/utils/data_load.py index c98347e..f629270 100644 --- a/utils/data_load.py +++ b/utils/data_load.py @@ -14,25 +14,16 @@ class DataLoadUtil: @staticmethod def load_exr_image(file_path): - # 打开 EXR 文件 exr_file = OpenEXR.InputFile(file_path) - - # 获取 EXR 文件的头部信息,包括尺寸 header = exr_file.header() dw = header['dataWindow'] width = dw.max.x - dw.min.x + 1 height = dw.max.y - dw.min.y + 1 - - # 定义通道,通常法线图像是 RGB float_channels = ['R', 'G', 'B'] - - # 读取 EXR 文件中的每个通道并转化为浮点数数组 img_data = [] for channel in float_channels: channel_data = exr_file.channel(channel, Imath.PixelType(Imath.PixelType.FLOAT)) img_data.append(np.frombuffer(channel_data, dtype=np.float32).reshape((height, width))) - - # 将各通道组合成一个 (height, width, 3) 的 RGB 图像 img = np.stack(img_data, axis=-1) return img diff --git a/utils/vis.py b/utils/vis.py index 831137f..66c5817 100644 --- a/utils/vis.py +++ b/utils/vis.py @@ -128,10 +128,10 @@ class visualizeUtil: if __name__ == "__main__": root = r"C:\Document\Local Project\nbv_rec\nbv_reconstruction\temp" model_dir = r"H:\\AI\\Datasets\\scaled_object_box_meshes" - scene = "test_obj" + scene = "box" output_dir = r"C:\Document\Local Project\nbv_rec\nbv_reconstruction\test" - visualizeUtil.save_all_cam_pos_and_cam_axis(root, scene, output_dir) + #visualizeUtil.save_all_cam_pos_and_cam_axis(root, scene, output_dir) visualizeUtil.save_all_combined_pts(root, scene, output_dir) visualizeUtil.save_target_mesh_at_world_space(root, model_dir, scene) #visualizeUtil.save_points_and_normals(root, scene,"10", output_dir, binocular=True) \ No newline at end of file