Lorenzo Lupo

EOFerror at the end of multiprocessing

8

**Describe the bug** Receiving EOFerror when multiprocessing, at the very end of training. **Minimal runnable code to reproduce the behavior** Launching a fairseq training with a simple transformer model on...

Truncation by tokenizer not working correctly

1

Hello, Truncation of the `input_ids` during tokenization, .i.e., [line 336](https://github.com/MilaNLProc/simple-generation/blob/73b760c60b76509390d286d4785fceeaa7d7fe8d/simple_generation/simple_generation.py#L336), does not work properly, throwing the following warning: > Asking to truncate to max_length but no maximum length is provided...