Multi GPU support with weight regularization

[look4pritam]

TensorFlow implementation of multi GPU treats weight regularization differently.

It computes partial losses and gradients on each GPU and then combines it on master CPU. It is similar to keras multi GPU.

In TensorFlow, weight regularization is not applied but each GPU. Master CPU computes weight regularization and add it final loss and gradients.

How this is implemented in keras multi GPU?