ERROR marie@32 Error in PSM_SPARSE_STEP : grid_sampler_2d_cpu not implemented for Half

[gregbugaj]

Need to handle this better. ref : 215000470

ERROR  marie@32 Error in PSM_SPARSE_STEP : grid_sampler_2d_cpu not implemented for Half


INFO   marie@31 Batch time [2, 101]: 1.0219602584838867                                                                                                          [10/03/24 10:19:31]
UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants. [checkpoint.py:464]
UserWarning: None of the inputs have requires_grad=True. Gradients will be None [checkpoint.py:91]
| WARNING: ran out of memory
WARNI… marie@32 OOM detected, clearing cache and retrying                                                                                                        [10/03/24 10:19:31]
ERROR  marie@32 Error in PSM_SPARSE_STEP : CUDA out of memory. Tried to allocate 1.34 GiB. GPU                                                                   [10/03/24 10:19:31]
| WARNING: ran out of memory
WARNI… marie@32 OOM detected, clearing cache and retrying                                                                                                        [10/03/24 10:19:32]
ERROR  marie@32 Error in PSM_SPARSE_STEP : CUDA out of memory. Tried to allocate 1.34 GiB. GPU                                                                   [10/03/24 10:19:32]
INFO   marie@32 Refinement step 1 : No change in image                                                                                                                              
WARNI… marie@32 Empty bounding boxes, possibly a blank page                                                                                                      [10/03/24 10:19:32]
INFO   marie@31 Batch time [3, 42]: 0.4753885269165039                                                                                                           [10/03/24 10:19:32]
INFO   marie@31 ICR Time elapsed: 3.549083709716797                                                                                                                                 
ERROR  marie@32 Error in PSM_SPARSE_STEP : grid_sampler_2d_cpu not implemented for Half                                                                                             
Payload: