RuntimeError: shape '[-1, 2, 34]' is invalid for input of size 61710

[jb33k]

I'm playing around with this wonderful code but I'm running into a curious issue when I try to train the model with my own data.

I replicated the personachat_self_original.json file structure and added my own data. I deleted dataset_cache_OpenAIGPTTokenizer file but when I try to train, I get this error:

INFO:train.py:Pad inputs and convert to Tensor
Traceback (most recent call last):
  File "train.py", line 252, in <module>
    train()
  File "train.py", line 164, in train
    train_loader, val_loader, train_sampler, valid_sampler = get_data_loaders(args, tokenizer)
  File "train.py", line 97, in get_data_loaders
    tensor = tensor.view((-1, datasets[dataset_name]["n_candidates"]) + tensor.shape[1:])
RuntimeError: shape '[-1, 2, 34]' is invalid for input of size 61710

I have triple checked that my dataset follows the same structure but I can't figure out why the training script doesn't like it.

Any ideas why this is happening?

[michelleful]

Same thing happens to me. Did you ever wind up solving this, @jb33k?

[michelleful]

For anyone else who may have this problem, my issue was that for some utterances in my data, the number of candidates for that utterance was less than n_candidates, but this line (currently #79 in train.py) assumes that you have at least that many:

 for j, candidate in enumerate(utterance["candidates"][-num_candidates:]):

As a quick fix, you can add a guard before this line to ensure that you're actually meeting that requirement.

[daspartho]

Is this still open? I can work on it!