Evaluation degradation w.r.t runtime and memory usage

[eslambakr]

Dears,

Thank you very much for your amazing work! I have one question regarding the evaluation performance, I noticed that the evaluation step takes almost the same time as training one epoch, which is weird for me. Also, I noticed the GPU memory also is reserved with the same amount during both phases, i.e., training and testing. Thus, I feel like the evaluation loop should be optimized more. I tried to optimize it but unfortunately, I couldn't find the root cause of this degradation in the runtime performance.

Thanks in advance!