Poor Genre results on demo images when compiled with Cuda10

[shubham-goel]

Hi,

I'm trying to run Genre on a machine with CUDA10 + 2080Tis (gencode arch 75). As far as I know, CUDA 9 cannot compile for gencode arch 75 hardware targets.

Therefore, I setup Genre in a singularity container containing pytorch0.5.0a + Cuda10.0 like this. This commit shows the minimal changes I had to make to the toolbox/*/setup.sh files.

Now, Genre (scripts/test_genre.sh) runs without any errors but the reconstructed 3D shapes look very poor. Do you have any idea why this would happen?

Here is the generated output/test_genre_full_model directory. Visualizations of input, predicted meshes:

[ztzhang]

Thanks for bringing up the issue. I'm under the impression that pytorch 0.4.1 does not support cuda10 and I can't find a pytorch 0.5.0a version. Is 0.5.0a build from source, according to this issue?

[xiumingzhang]

Hi, could it be because you had too high of a threshold in visualizing the voxels?

[shubham-goel]

Shouldn't be a voxel threshold issue because I'm using the default threshold for standard demo images by running scripts/test_grenre.sh.

Re: pytorch version, you're right that pytorch 0.4.1 does not officially provide any prebuilt binaries for cuda10. But it should still be possible to compile pytorch 0.4.1 from source with Cuda10. This nvidia container that I'm using does exactly that! I was confused why the torch.__version__ was 0.5.0a in the container but that issue explains it.

[ztzhang]

In this case, it seems the only way for us to trace this issue is by reproducing the results you have here with that specific environment. I'll try to find sometime this weekend to look into this issue.

[ztzhang]

@shubham-goel I spent some time setting up the environment on our cluster, but haven't gone through the testing. I'll ping in this thread when I have some updates.

[shubham-goel]

Thanks @ztzhang! In interest of time, I setup Genre on a labmate's old machine (with CUDA9/1080Ti) yesterday -- so this issue isn't time-critical for me anymore. You may delay working on it, or even close it if you want.

But since the solution to this issue will probably provide a way of running Genre with CUDA10 on the newer-architecture GPUs, it'd still be useful to many others out there! I leave the decision to you. Thanks again!!