Experiment result is not consistent with the result reported on the paper

[lijing1996]

Hi, thanks a lot for sharing your work! I download your full approach model pretrained on the panoContext dataset and use it to predict on the test set of the panoContext dataset. However, I find that the results inferred by your pretrained model are 73.85, 1.07 and 3.40 respectively while the results on the paper are 74.48, 1.06 and 3.34 respectively. I doubt wheter the pretrained model can obtain the results reported on the paper or whether I did something wrong... Can u help me? Thanks a lot!

[zouchuhang]

@lijing1996 The optimization step is sampling-based and the solver is L-BFGS based, therefore it allows a slight quantitative fluctuation in prediction, as long as you use the correct full pre-trained model. Your testing results looks promising to me but I'm not sure about your optimization step details. On the other hand if you want to try gradient ascent you can check the code by https://github.com/sunset1995/pytorch-layoutnet, which shows a slightly better quantitative performance.

[bertjiazheng]

We use panoOptimization with default hype-parameters. Do we need to tune any hyper-parameters?

[zouchuhang]

@bertjiazheng I believe this is the quantitative performance fluctuation by using L-BFGS (quoted below). You can change the parameters in pano_line_solver.m to have longer iterations and smaller threshold of stopping criteria to have better solution, while you'll get longer time to solve. If you'd prefer a gradient ascent based you can play around with the code as I suggested below.

@lijing1996 The optimization step is sampling-based and the solver is L-BFGS based, therefore it allows a slight quantitative fluctuation in prediction, as long as you use the correct full pre-trained model. Your testing results looks promising to me but I'm not sure about your optimization step details. On the other hand if you want to try gradient ascent you can check the code by https://github.com/sunset1995/pytorch-layoutnet, which shows a slightly better quantitative performance.