Multiple GPUs using DataParallel

[nerkulec]

This allows for utilization of multiple GPUs for training of MALA models. It is done with DataParallel, which in contrast to DistributedDataParallel does not require multiprocessing. Simply use

parameters.running.num_gpus = 4

No additional changes to python or slurm scripts are needed.

[RandomDefaultUser]

Thanks for the PR! I really like this and I think we should implement this, before we move to DDP (which is the next thing on my todo list), so I just had a look at this PR. I have two suggestions/adjustments:

1. I would get rid of the example, since it only showcases one changed parameter. We can simply update the documentation.
2. I think the functionality of parameters.running.num_gpus = 4 can simply be absorbed into parameters.use_gpu. That value could simply be an int instead of a bool, without any drawbacks.

I have made these changes in a PR here: https://github.com/nerkulec/mala/pull/1, if that looks OK we could first merge that PR and then this one here.

[RandomDefaultUser]

Oh wait, there is one potential problem, and that is when training with multiple GPUs and then loading to run with either only one GPU or MPI+GPU. I will test this right away!

[nerkulec]

I like your changes :) I added a small fix to my part. Now the checks make much more sense. Feel free to merge when you resolve the potential problem you mentioned.

[RandomDefaultUser]

I confirmed that inference pipeline indeed still works!

[RandomDefaultUser]

In theory this works, once this is benchmarked we can merge it!

[nerkulec]

The benchmarks showed that training on multi-gpu using DataParallel was slower than with one gpu. I'm closing this since we have the DDP implementation.