tensorflow-wavenet
tensorflow-wavenet copied to clipboard
ibab
Improve GPU performance
The performance of the network on GPUs seems to be lagging behind the CPU performance. I suspect that this is because the 2D convolution isn't designed to work efficiently if the height of the input is 1. It shouldn't be too difficult to write some custom code to perform an efficient 1D convolution. For example, fft could be used for this.
Just compared GPU and CPU again, and this time training is significantly faster on the GPU.
We would probably be able to train larger networks (and with larger sample sizes) if we had a single C++ op that does causal 1d convolution, as we would be able to avoid all the extra copying that's currently being done. But writing and maintaining the op might be a pain :/
I've done it before, but not with gpu support...
On Thu, Oct 6, 2016, 18:58 Igor Babuschkin [email protected] wrote:
We would probably be able to train larger networks (and with larger sample sizes) if we had a single C++ op that does causal 1d convolution, as we would be able to avoid all the extra copying that's currently being done. But writing and maintaining the op might be a pain :/
— You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/ibab/tensorflow-wavenet/issues/25#issuecomment-252111726, or mute the thread https://github.com/notifications/unsubscribe-auth/ADCF5gay3mfBQ0wKyF2mamL0xFbu5kq9ks5qxX0mgaJpZM4J_k-r .
@ibab I've made a pull request for mixed precision training https://github.com/ibab/tensorflow-wavenet/pull/384
and would like to encourage the authors and community to give it a try :)
