How parallel is Hyperas?

[myhussien]

Hi, and thanks for the nice library.

Is it possible to run hyperas on multiple GPU's? if so, would you mind giving a small example?

Thanks!

[myhussien]

So, keras allows for parallel GPUs, but I'm not sure how to make the trials parallel as well. Any help would be appreciated. @dafnevk can you shed some light on this issue please?

[maxpumperla]

@myhussien sorry for the late reply. hyperopt underneath hyperas has a scale-out module using mongodb, which you can read more about here. If your mongo workers all have access to multiple GPUs and keras is installed on them, then they'll be used. If you do this locally on one machine this is unlikely to bring any benefit, unless you can dedicate one GPU per mongo worker (not that simple to set up, would have to dig into mongo docs).

I think the prototypical situation is that you utilise multiple CPU threads in parallel search with hyperopt.

If you train two keras models on the same machine at the same time, will they effectively share access to your GPUs? If so, it might be worth giving this a shot.

[TBomberman]

I made multiple GPUs visible using CUDA_VISIBLE_DEVICES. However, it's just as slow as when it was running on 1 GPU. I also only see one model being trained at a time.'

Anyone know why it's not running multiple GPUs?

[SachitNayak]

Any updates here? Parallel trials would be fantastic!

[JonnoFTW]

I did write a script that can run hyperas in distributed mode: https://github.com/maxpumperla/hyperas/blob/master/examples/mnist_distributed.py and https://github.com/maxpumperla/hyperas#running-hyperas-in-parallel

My plan of attack would be to run it using multiple distributed workers on a single machine (start as many workers as you have GPUs). Have the job select the GPU with the smallest current load (or wait until one is free).

[SachitNayak]

Thank you so much! I haven't tried your solution yet. Nonetheless, thank you very much.