dopamine icon indicating copy to clipboard operation
dopamine copied to clipboard

setting tf_device = '/gpu:1' mallocs all memory on /gpu:0

Open Chazzz opened this issue 6 years ago • 0 comments

I ran into a nasty memory usage error, which dopamine is already fairly well structured to take care of. The title is fairly self-explanatory, and running a simultaneous job with tf_device = '/gpu:0' will fail if the gpu:1 job was started first. I haven't done any analysis on how this impacts performance, but setting allow_growth=True in the gpu_options for the session initialized in run_experiment would do the trick. Alternately, setting which gpu to use in the gpu_options (by making it the only one visible) would also work but require Runner to be configured in gin to set the correct gpu, and also needs the handle the nasty cpu-only case, so I don't like that fix due to the complexity of the change.

Chazzz avatar Feb 10 '19 10:02 Chazzz