Missing the evaluation for fused depth

[xjixzz]

Hi,

Thank you for sharing your project! It is really amazing.

It seems that the evaluate_depth.py provides the evaluation for mono depth and mvs depth, and upper bound results, but lacks the evaluation for fused depth. I tried to add the evaluation code for fused depth only to find that fused depth is worse than mvs depth with my evaluation code. The test results with your pretrained weights are as follows:

mono results: abs_rel | sq_rel | rmse | rmse_log | a1 | a2 | a3 | & 0.113 & 0.823 & 4.724 & 0.190 & 0.879 & 0.960 & 0.981 \

mvs results: abs_rel | sq_rel | rmse | rmse_log | a1 | a2 | a3 | & 0.094 & 0.704 & 4.389 & 0.175 & 0.902 & 0.965 & 0.983 \

fuse results: abs_rel | sq_rel | rmse | rmse_log | a1 | a2 | a3 | & 0.112 & 0.818 & 4.709 & 0.190 & 0.879 & 0.960 & 0.982 \

upbound results: abs_rel | sq_rel | rmse | rmse_log | a1 | a2 | a3 | & 0.082 & 0.618 & 4.135 & 0.162 & 0.915 & 0.969 & 0.985 \

I also tested my own trained model, the results are as follows: mono results: abs_rel | sq_rel | rmse | rmse_log | a1 | a2 | a3 | & 0.116 & 0.920 & 4.868 & 0.193 & 0.874 & 0.959 & 0.981 \

mvs results: abs_rel | sq_rel | rmse | rmse_log | a1 | a2 | a3 | & 0.097 & 0.785 & 4.512 & 0.177 & 0.899 & 0.963 & 0.982 \

fuse results: abs_rel | sq_rel | rmse | rmse_log | a1 | a2 | a3 | & 0.098 & 0.788 & 4.518 & 0.177 & 0.899 & 0.963 & 0.982 \

upbound results: abs_rel | sq_rel | rmse | rmse_log | a1 | a2 | a3 | & 0.086 & 0.698 & 4.269 & 0.165 & 0.911 & 0.968 & 0.984 \

I think there must be something wrong with my modification of the evaluation code. Could you please update the evaluation code to test fused depth?

[youtufan]

I also encountered the same problem. Is this problem solved?

[xjixzz]

Not yet

[RuijieZhu94]

I have the same question. It seems that the mask for fused depth did not work on the evaluation. And the fused depth is always worse than the mvs depth.

[Adam-Zha]

Is this operation even reasonable? If I can conduct tests in this manner, I could churn out ten top-tier conference papers in a year LOL.

[logicred]

I have the same problem. Since there is no reply by author from Sep 26, 2022 to now, I guess that something is wrong in paper or in codes.

[CydKnocking]

Hi @xjixzz , could you please provide the revised version of evaluate_depth.py? I encountered the same problem with you. But sadly I did not managed to reproduce the same "fuse results" with yours. Thanks a lot!

[xjixzz]

Here is my revised evaluate_depth.py @CydKnocking

[CydKnocking]

@xjixzz Thank you for your code! I got similar results with yours when tested on my own trained model.

[fanrz]

Has anyone solved the problem on fused results?

[lhx0416]

The author mentioned in the article that the trust mask seems to have no effect. In the training logs, it's observable that the MVS depth map and the fused depth map are identical, suggesting there might be some issues within the program.