TensorNet
TensorNet copied to clipboard
Bihaqo
training speed
Hello, has TensorNet accelerated during training? Hope you can share the results of the training speed, thank you
In my experience, tensor net topically don't help with the training speed (unless you're going for truly gigantic layers, which would be impractical without tensornet): the time per iteration is usually almost the same (maybe like 10% faster) and you need to do a bit more iterations than when training without tensornet. That's probably because on training a single matmul is rarely the bottleneck of the entire training process.
On Thu, 16 Apr 2020 at 07:16, WWWJL [email protected] wrote:
Hello, has TensorNet accelerated during training? Hope you can share the results of the training speed, thank you
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Bihaqo/TensorNet/issues/7, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABK6V2OD2MDGUCXQLU6T76DRM2PDTANCNFSM4MJHV4JQ .
Also note that in the MATLAB implementation I wrote the backward pass manually and it's much slower than the speed you can get with autodiff (e.g. by writing it in tensorflow or pytorch)
On Thu, 16 Apr 2020 at 13:52, Alexander Novikov [email protected] wrote:
In my experience, tensor net topically don't help with the training speed (unless you're going for truly gigantic layers, which would be impractical without tensornet): the time per iteration is usually almost the same (maybe like 10% faster) and you need to do a bit more iterations than when training without tensornet. That's probably because on training a single matmul is rarely the bottleneck of the entire training process.
On Thu, 16 Apr 2020 at 07:16, WWWJL [email protected] wrote:
Hello, has TensorNet accelerated during training? Hope you can share the results of the training speed, thank you
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Bihaqo/TensorNet/issues/7, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABK6V2OD2MDGUCXQLU6T76DRM2PDTANCNFSM4MJHV4JQ .