[Tune] Ray Tune gives incorrect warning about not using GPUs when used with Ray Lightning

[yinweisu]

When trained on gpu, I see the following warning:

2022-05-03 00:02:20,033 WARNING tune.py:637 -- Tune detects GPUs, but no trials are using GPUs. To enable trials to use GPUs, set tune.run(resources_per_trial={'gpu': 1}...) which allows Tune to expose 1 GPU to each trial. You can also override `Trainable.default_resource_request` if using the Trainable API.

To reproduce, simply run the ray tune example code with gpu.

I think this is related to the usage of get_tune_resources or the fact that the head process is cpu only. Either way, this warning is incorrectly triggered.

Also, is there a way to disable such warning? I tried to filter it with warnings.filterwarnings but wasn't able to. My assumption is that ray would overwrite the warning settings underneath.

[xwjiang2010]

I didn't get a chance to try out the GPU setup myself. But from what I can tell reading the code, this should not be expected. Could you maybe print out get_tune_resources and follow through the logic here? I think it should land in Line 661. It may be interested to see why it doesn't in your case.

As for warnings.filterwarnings - it won't work for logger.warn. See https://stackoverflow.com/questions/66727458/warnings-filterwarningsignore-doesnt-ignore-logging-warning. Maybe you could try setLevel? But I agree that we should have an easy way to configure it.

cc @Yard1 along the lines of what you brought up today! We should slot this in.

[yinweisu]

OK. Just took a look at the source code for ray 1.12: https://github.com/ray-project/ray/blob/ray-1.12.0/python/ray/tune/tune.py#L629 Apparently this issue has been solved in master branch. May I know when the next release will come?

[xwjiang2010]

@avnishn - 1.13 release manager.

[avnishn]

Our current time table has us releasing around the 20th of may.

You can install the nightly version of ray by calling ray install nightly in your terminal, after installing ray, if you need a sooner release. We are also release ray 1.12.1 soon. cc @amogkam