MemoryError [solved]

[neoctbits]

When run data_loader.py, i got some problem:

[TL] Vocabulary from /home/lab/Desktop/text-to-image-master/vocab.txt : <S> </S> <UNK> vocabulary with 5430 words (includes start_word, end_word, unk_word) start_id: 1 end_id: 2 unk_id: 5429 pad_id: 0

• tokenized 81890 captions img_capt: this flower has bright purple spiky petals and greenish sepals below them nltk.tokenize.word_tokenize(img_capt): ['this', 'flower', 'has', 'bright', 'purple', 'spiky', 'petals', 'and', 'greenish', 'sepals', 'below', 'them'] img_capt_ids: [6, 3, 7, 31, 18, 165, 4, 5, 319, 83, 374, 120] id_to_word: ['this', 'flower', 'has', 'bright', 'purple', 'spiky', 'petals', 'and', 'greenish', 'sepals', 'below', 'them']
• 8189 images found, start loading and resizing ...
• loading and resizing took 77.8853452205658s n_captions: 81890 n_images: 8189 n_captions_per_image: 10 n_images_train:8000 n_captions_train:80000 n_images_test:189 n_captions_test:1890 Traceback (most recent call last): File "data_loader.py", line 168, in save_all((images_train_256, images_train), '_image_train.pickle') File "data_loader.py", line 165, in save_all pickle.dump(targets, f) MemoryError

[zsdonghao]

it is MemoryError... you may need to solve it yourself

[neoctbits]

Thanks@zsdonghao，i had resolve the problem using dill module instead of pickle. but it also memery costly. The alternative way is merge data_ loader.py into train_txt2img.py to get rid of pickle.

[zsdonghao]

cool~ thank you I will mark it done~

[qpwo]

@boluanace can you post a GitHub gist of your merged data_loader.py and train_txt2img.py please? I am getting the memory error and it is killing me