Question about validation

[ChengkaiYang]

Dear Author: I hope this issue finds you well! First of all, your work is constructive. When i apply your method to my work, i have found a puzzled phenomenon. In FDE, training losses are very low but validation or test results are not accurate as training(quickly get worse after training 2 epochs). I guess it is because the KDE smoothing isn't conducted when validating but only training. I tried to exposure validation labels(should not be seen when validation) on purpose and smoothed the feature space during validation as well and got almost ideal result as training results. According to that, although we could not see labels when validating or testing, I wonder whether it is effective to deal similar smoothing operator in validating or testing. But i have double checked your codes and i have not found you smoothed feature space in validation. What caused my problem? I have some speculations. 1)The decoder used for your work is only a MLP regressor. But in my work, the decoder has 3 MLPs and 2 RELUs. That may cause a phenomenon that features belonging to the neighbour bins might not have the similar feature measured by Euclidean distance. So the distance between smoothed and not smoothed can be very large. 2)The feature updated so violently. Although I have conducted smoothing operator, it can't learn effective informations. 3)Some bins are empty, when smoothing we use initialized zero vector to smooth. That may destory the distribution of the value and may got incorrect regression results.Is it okay to use zero vector representing empty bins to smooth when training? Also, i find a possible more correct sequece for running the code. Why FDS.update_running_stats is after FDS.update_last_epoch_stats? if FDS.update_running_stats is after FDS.update_last_epoch_stats, it leads model uses smoothed running statistics the epoch before last epoch but not the running statistics updated last epoch. So if it is correct to change the sequence of FDS.update_running_stats and FDS.update_last_epoch_stats? Looking forward for your reply.Thanks!