How can I export textured mesh?

[wukailu]

The exported .obj file contains only the shape, not the corresponding material and color.

[bennyguo]

Hi, I haven't write code about this. But you could achieve this by (1) query the color (at a fixed viewing direction) for each mesh vertex; (2) rewrite the obj saving code to support color attributes. I'll add this feature when I have time.

[adithya1-7]

@bennyguo I was not able to figure this part out. Can you tell us which python files have to be modified to achieve the required output? It would be ideal if you provide the code modification snippets with line number. Thanks in advance

[iraj465]

@bennyguo getting textured mesh would be great! Can you please help on this?

[adithya1-7]

@bennyguo Found this snippet for colour extraction, but don't know how it applies to current implementation ray_proj = torch.from_numpy(vertices).cuda().float().split(8192 * 32, 0) vertex_col = [model.mesh_color_forward(ray) for ray in ray_proj] vertex_col = (torch.concat(vertex_col).cpu().detach().numpy() * 255.).astype( np.uint8) mesh = trimesh.Trimesh(vertices, triangles, vertex_colors=vertex_col)

[iraj465]

I was getting some idea from this: https://github.com/xrr-233/Textured-NeuS

But not being able to implement those functions in this repo.

[bennyguo]

Hi guys, I just implemented the textured mesh export, please have a try! If you are resuming from old checkpoints, remember to add export.chunk_size=2097152 export.export_vertex_color=true after your command. For NeRF, you may need a higher marching cubes resolution (~512) to get better vertex color predictions.

Here's a textured chair extracted from NeuS:

[iraj465]

Getting this error. I turned on the flag for export mesh colour. This was working on the last commit. (just the mesh)

[bennyguo]

@iraj465 I also encountered this problem when viewing the mesh using MeshLab, but it seems just a warning. It may be related to using Trimesh to save the mesh. I'll look into it later today.

[iraj465]

Also, no mesh is coming in obj. Only a bunch of points, whereas it is being trained (in iters images)

Mesh obj:

Iters sample image:

[bennyguo]

Are you training NeRF model? You need to tune model.geometry.isosurface.threshold to get reasonable meshes (in your case set to a smaller value).

[iraj465]

Yes, i'm training nerf.

This is what i'm getting for the above shoe now.

Earlier i was getting this for the same threshold (in last commit):?

[bennyguo]

Any luck trying a lower threshold?

[iraj465]

Yeah, lowered the threshold from 5.0 to 3.0. Still the whole shoe is not coming, also the texture seems kinda off.

Image samples. <img width="1067" alt="Screenshot 2023-04-05 at 6 21 02 PM" src="https://user-images.githubusercon tent.com/32464366/230085818-1f9aad21-9fde-41df-a6ee-ce3a3a426d1f.png">

Config used:

    radius: ${model.radius}
    feature_dim: 16
    density_activation: trunc_exp
    density_bias: -1
    isosurface:
      method: mc
      resolution: 1024
      chunk: 2097152
      threshold: 3.0
    xyz_encoding_config:
      otype: HashGrid
      n_levels: 16
      n_features_per_level: 2
      log2_hashmap_size: 19
      base_resolution: 16
      per_level_scale: 1.447269237440378  
    mlp_network_config:
      otype: FullyFusedMLP
      activation: ReLU
      output_activation: none
      n_neurons: 64
      n_hidden_layers: 1
  texture:
    name: volume-radiance
    input_feature_dim: ${model.geometry.feature_dim}
    dir_encoding_config:
      otype: SphericalHarmonics
      degree: 4    
    mlp_network_config:
      otype: FullyFusedMLP
      activation: ReLU
      output_activation: Sigmoid
      n_neurons: 64
      n_hidden_layers: 2
  
system:
  name: nerf-system
  loss:
    lambda_rgb: 1.
    lambda_distortion: 0.001
  optimizer:
    name: Adam
    args:
      lr: 0.01
      betas: [0.9, 0.99]
      eps: 1.e-15
  scheduler:
    name: MultiStepLR
    interval: step
    args:
      milestones: [10000, 15000, 18000]
      gamma: 0.33

checkpoint:
  save_top_k: -1
  every_n_train_steps: ${trainer.max_steps}

export:
  chunk_size: 2097152
  export_vertex_color: True 

trainer:
  max_steps: 50000
  log_every_n_steps: 200
  num_sanity_val_steps: 0
  val_check_interval: 5000
  limit_train_batches: 1.0
  limit_val_batches: 2
  enable_progress_bar: true 
  precision: 16

[bennyguo]

@iraj465 I think this is related to the new scene center estimation method mentioned in https://github.com/bennyguo/instant-nsr-pl/issues/55#issuecomment-1499944616. In your case it seems the scene center is biased, and your object is not fully wrapped by the bounding box. I'll push a fix later today.

[iraj465]

I see, the poses are not transformed relatively so the whole scene is shifted. Yeah @bennyguo the fix push would be great. I have also outlined this problem in this issue if you need further details or some testing to do. Issue #60

[bennyguo]

Camera normalization fixed in the latest commit.

[bennyguo]

Camera normalization fixed in the latest commit.