Chapter 15.4. Pretraining word2vec: AttributeError: Can't pickle local object 'load_data_ptb.<locals>.PTBDataset'

[keyuchen21]

AttributeError: Can't pickle local object 'load_data_ptb..PTBDataset'

can anyone help with this error?

[HJL-echoandy]

这个问题搞定了了吗 请问

[cx-olquinjica]

@HJL-echoandy to anyone who is facing this problem I sort of found a quick fix.... After struggling with this same issue on a MacBook Pro M2, according to what I found online, this error message was mostly related to how Jupyter handles multithreading, particularly pickle.

So this solution worked for me:

1. Create a python file in the same folder where you have your word_embedding_dataset.ipynb file, let's say you'll call it ptb_dataset.py
2. Move (i.e., delete from the word_embedding_dataset.ipynb) the class PTBDataset(torch.utils.data.Dataset), and the function def batchify. Move them both to the ptb_dataset.py
3. Import the class PTBDataset and the function batchify now into the word_embedding_dataset.ipynb (i.e., from ptb_dataset import PTBDataset, batchify
4. Run the notebook again, and you shall be fine.

Note: You can refer to my notes in case of doubt.

Another thing to pay attention is that the only way I found to deal with this issue was to combine both lesson 15.3 and 15.4 in one notebook (i.e., dataset_word_embedding.ipynb), because I defined load_data_ptb locally.