ray_lightning icon indicating copy to clipboard operation
ray_lightning copied to clipboard

Cannot Use GPUStatsMonitor callback with Ray Lightning

Open DavidMChan opened this issue 2 years ago • 1 comments

The GPUStatsMonitor Callback records information about the GPU utilization in Tensorboard logs, however when running with ray_lightning, it raises a MisconfigurationException:

pytorch_lightning.utilities.exceptions.MisconfigurationException: You are using GPUStatsMonitor but are not running on GPU since gpus attribute in Trainer is set to None.

This is due to the code in the stats monitor callback:

if trainer._device_type != DeviceType.GPU:
            raise MisconfigurationException(
                "You are using GPUStatsMonitor but are not running on GPU"
                f" since gpus attribute in Trainer is set to {trainer.gpus}."
            )

It seems like ray_lightning, thus, doesn't set the DeviceType to GPU - which may have other unintended consequences later on.

This may also be solved by #118, but It's not entirely clear

DavidMChan avatar Feb 22 '22 00:02 DavidMChan