Issues with multi target problem

[Vergangenheit]

Hello Sir,

I was wondering if you could help me with this. I was trying to run a multi-target multi-variate experiment using "DecoderTransformer" model.

So I have a configuration in "dataset_params" as such:

"target_col": ['UP_MPNTLCDMRN_1', 'UP_MPNTLCSMBC_1', 'UP_PEPIZZA_1',
                       'UP_PRCLCDMZRD_1', 'UP_PRCLCDPLRM_1',
                       'UP_PRCLCDPRZZ_1', 'UP_PRCLCMINEO_1'],
 "relevant_cols": [col for col in df.columns[1:-2] if 'wind_gusts_100m' not in col]

with 7 targets and about 88 covariates, resulting in 95 relevant columns.

What happens running train_function is that at compute_validation, the result data before inverse scaling is (forecast_length, batch_size). But the StandardScaler._scaler is of (len(target_col), ) size.

What am I getting wrong? Is it actually not configured for multi target problems?

Thank you very much

Lorenzo Ostano

[isaacmg]

Are you including "n_targets" in the parameters (general parameters)? That needs to be included as well? See https://github.com/AIStream-Peelout/flow-forecast/blob/5672d4540fe73fbc4165bc2c4d3b0131d0e4a169/tests/multitask_decoder.json

[Vergangenheit]

I just added it. And now the issue is in the compute_loss of the forward pass on the training set. I've got a label of shape (batch_size, forecast_length, 1, n_targets), while the outputs are (batch_size, forecast_length)...

Here's the full configuration dictionary:

config_default={
      "model_name": "DecoderTransformer",
      "model_type": "PyTorch",
      "model_params": {
      "n_time_series":95,
      "n_head": 8,
      "forecast_history":168,
      "n_embd": 1, 
      "num_layer": 5,
      "dropout":0.1,
      "q_len": 1,
      "scale_att": False,
      "forecast_length": 12, 
      "additional_params":{}
     },
     "n_targets": 7,
     "dataset_params":
     {
         "class": "default",
          "training_path": file_path,
          "validation_path": file_path,
          "test_path": file_path,
          "batch_size":64,
          "forecast_history":168,
          "forecast_length":12,
          "train_end": int(train_number),
          "valid_start":int(train_number+1),
          "valid_end": int(validation_number),
          "target_col": ['UP_MPNTLCDMRN_1', 'UP_MPNTLCSMBC_1', 'UP_PEPIZZA_1',
                       'UP_PRCLCDMZRD_1', 'UP_PRCLCDPLRM_1',
                       'UP_PRCLCDPRZZ_1', 'UP_PRCLCMINEO_1'],
          "relevant_cols": [col for col in df.columns[1:-2] if 'wind_gusts_100m' not in col],
          "scaler": "StandardScaler", 
          "interpolate": False,
          "sort_column":"time",
     },
     "training_params":
      {
        "criterion":"DilateLoss",
        "optimizer": "Adam",
        "optim_params":
        {
        },
        "lr": 0.001,
        "epochs": 4,
        "batch_size":64
      },
      "early_stopping": {
          "patience":3
      },
      "GCS": False,
      "sweep":False,
      "wandb":False,
      "forward_params":{},
      "metrics":["DilateLoss"],
      "inference_params":
        {     
              "datetime_start":"2020-08-06 23:00:00",
                "hours_to_forecast": 1344, 
                "test_csv_path":file_path,
                "decoder_params":{
                    "decoder_function": "simple_decode", 
                  "unsqueeze_dim": 1
                },
                "dataset_params":{
                  "file_path": file_path,
                  "forecast_history":168,
                  "forecast_length":12,
                  "relevant_cols": [col for col in df.columns[1:-2] if 'wind_gusts_100m' not in col],
                  "target_col": ['UP_MPNTLCDMRN_1', 'UP_MPNTLCSMBC_1', 'UP_PEPIZZA_1',
                               'UP_PRCLCDMZRD_1', 'UP_PRCLCDPLRM_1',
                               'UP_PRCLCDPRZZ_1', 'UP_PRCLCMINEO_1'],
                  "scaling": "StandardScaler",
                  "interpolate_param": False,
                }
          },
    }

[isaacmg]

Sorry for the slow response (been away). You might want to remove unsqueeze dim from the params.

[Vergangenheit]

No problem for the slow response. Look let me keep it opened for few more days. I don't think I'm unsqueezing tensors' dimension during training. Let me come back to you in few days, I understand this is an open-source in-construction library and I don't want to take up your time away from your roadmap....

[isaacmg]

Were you able to fix the issue? If you not I'll investigate further. It may be a bug with the framework.

[Vergangenheit]

Hi Isaac.

Unfortunately not. I removed the unsqueeze parameter but the loss function computation still doesn't work.

/content/flow_forecast/flood_forecast/pytorch_training.py in compute_loss(labels, output, src, criterion, validation_dataset, probabilistic, output_std, m)
    285         loss: Criterion = criterion(labels.float(), output, src, m)
    286     else:
--> 287         assert len(labels.shape) == len(output.shape)
    288         assert labels.shape[0] == output.shape[0]
    289         loss: Criterion = criterion(output, labels.float())

[isaacmg]

Okay I'll run some additional testing. I believe I've seen something similar before. But don't remember the way I fixed it.

[isaacmg]

Never mind I just remembered that I don't believe at the moment DecoderTransformer works with multiple targets. So this is a possible enhancement not a bug. Basically just requires adding another linear layer.

[theoctopusride]

@isaacmg is there an up-to-date list of flow-forecast models that currently work with multiple targets? I've been struggling with the pytorch-forecasting library to get their models working with multiple targets.

[isaacmg]

@theoctopusride The following models support multiple targets CustomTransformerDecoder Informer MultiHeadSimple