Slow when train on GPU

[dangne]

I'm using this RBF implementation to train on top of an CuDNNLSTM with GPU. However, I got a tremendous decrease in performance when compare to training the CuDNNLSTM alone. My guess is because of this RBF layer does not support GPU so it is the bottle-neck of the model.

[PetraVidnerova]

I have to take a close look on this. But I think the computations of RBF layer cannot benefit from GPU as well as other layers. The second thing is what part of it is done on GPU if any. But it definitely will be the bottle-neck of the model.