ray_lightning
ray_lightning copied to clipboard
Cannot Use GPUStatsMonitor callback with Ray Lightning
The GPUStatsMonitor Callback records information about the GPU utilization in Tensorboard logs, however when running with ray_lightning, it raises a MisconfigurationException:
pytorch_lightning.utilities.exceptions.MisconfigurationException: You are using GPUStatsMonitor but are not running on GPU since gpus attribute in Trainer is set to None.
This is due to the code in the stats monitor callback:
if trainer._device_type != DeviceType.GPU:
raise MisconfigurationException(
"You are using GPUStatsMonitor but are not running on GPU"
f" since gpus attribute in Trainer is set to {trainer.gpus}."
)
It seems like ray_lightning, thus, doesn't set the DeviceType to GPU - which may have other unintended consequences later on.
This may also be solved by #118, but It's not entirely clear