pacnet
pacnet copied to clipboard
NVlabs
About "Native Implemention"
Hi Suhang! Thanks for your sharing the code! I just wonder why there is a native implemention to control the computational flow. Is there any theoretical or performance differences between the native implemantion and non-native implemention?
The native implementation is using only standard pytorch layers/operations, while the other one uses the Function interface at places. The latter can have some memory advantage for larger input sizes.
Hey Hang, thanks for the comment! Additionally I am wondering if you have any statistics on how big the performance gap is between native and functional implementations, especially in case of large input sizes, or any differences in backprop speed, because I did some basic benchmarking and did not find significant difference there. Thanks!
There shouldn't be any noticeable differences in speed. The non-native implementation has advantages in terms of peak memory usage, but I have no statistics regarding the gap. The native version was there mostly for debugging purposes, and the non-native version should be the preferred option now.
Ah i see. Thanks! The native version would also be useful for prototyping because it is easier to build upon.
Hey Hang, after a second thought, the original version is actually correct to me. kernel will be used when you compute grad_input, and input will be used when you compute grad_kernel. See the highlights below.
In your updated version I suppose you will see errors in https://github.com/NVlabs/pacnet/blob/master/pac.py#L352 that input is None when ctx.needs_input_grad[0] == False and ctx.needs_input_grad[0] == True. Am I right?
@Jerrypiglet That's right. I wasn't careful enough on this. The change is now reverted.