yaqlee

Why the training always fails on the full data sets?

17

When training with the full dataset, I always encounter one of the following two issues: 1. When training with a single worker, I get an error message > `(raylet) [2023-04-01...

Illegal Memory Access Error When Running Training on GPU Cluster

2

I wrote a feature builder and target builder and created a model class that inherits from torchModuleWrapper. I also wrote a new objective class based on my model. These components...

A RUNTIMEWARNING message always happen when run training, without interrupting the training process

7

It seems that it did not affect or interrupt the training process, but it looks too much: `/root/code/nuplan-devkit/nuplan/common/maps/nuplan_map/utils.py:413: RuntimeWarning: invalid value encountered in cast return elements.iloc[np.where(elements[column_label].to_numpy().astype(int) == int(desired_value))]` how can...