generator raised StopIteration when preprocessing when preprocessing

[voidism]

Hi, thank you for releasing your code! I ran the preprocessing code preprocess.py and meet a runtime error.

INFO:root:skip this step as /workspace/helo_word/data/conll2014 is NOT empty
INFO:root:STEP 0-8. Download language model
INFO:root:skip this step as /workspace/helo_word/data/language_model/data-bin is NOT empty
INFO:root:STEP 1. Word-tokenize the original files and merge them
INFO:root:STEP 1-1. gutenberg
INFO:root:skip this step as /workspace/helo_word/data/gutenberg/gutenberg.txt already exists
INFO:root:STEP 1-2. tatoeba
INFO:root:skip this step as /workspace/helo_word/data/tatoeba/tatoeba.txt already exists
INFO:root:STEP 1-3. wiki103
INFO:root:skip this step as /workspace/helo_word/data/wiki103/wiki103.txt already exists
INFO:root:STEP 2. Train bpe model
INFO:root:skip this step as /workspace/helo_word/data/bpe-model/gutenberg.model already exists
INFO:root:STEP 3. Split wi.dev into wi.dev.3k and wi.dev.1k
INFO:root:skip this step as /workspace/helo_word/data/bea19/wi+locness/m2/ABCN.dev.gold.bea19.3k.m2 already exists
INFO:root:STEP 4. Perturb and make parallel files
INFO:root:Track 1
INFO:root:STEP 4-1. writing perturbation scenario
INFO:root:STEP 4-2. gutenberg
# multiprocessing settings
# prepare inputs
# work
  0%|                                                                                        | 0/1 [00:00<?, ?it/s]
--- SKIP ---
  0%|                                                                                        | 0/1 [00:08<?, ?it/s]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/pattern3/text/__init__.py", line 412, in _read
    raise StopIteration
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/workspace/helo_word/gec/perturb.py", line 160, in make_parallel
    perturbation = apply_perturbation(words, word2ptbs, word_change_prob, type_change_prob)
  File "/workspace/helo_word/gec/perturb.py", line 121, in apply_perturbation
    w = change_type(w, t, type_change_prob)
  File "/workspace/helo_word/gec/perturb.py", line 34, in change_type
    word = conjugate(word, verb_type)
  File "/opt/conda/lib/python3.7/site-packages/pattern3/text/__init__.py", line 2123, in conjugate
    b = self.lemma(verb, parse=kwargs.get("parse", True))
  File "/opt/conda/lib/python3.7/site-packages/pattern3/text/__init__.py", line 2088, in lemma
    self.load()
  File "/opt/conda/lib/python3.7/site-packages/pattern3/text/__init__.py", line 2042, in load
    for v in _read(self._path):
RuntimeError: generator raised StopIteration
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "preprocess.py", line 169, in <module>
    args.word_change_prob, args.type_change_prob))
  File "preprocess.py", line 15, in maybe_do
    func(*inputs)
  File "/workspace/helo_word/gec/perturb.py", line 183, in do
    p.map(make_parallel, inputs_li)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
RuntimeError: generator raised StopIteration

I tried to skip processing gutenberg corpus, but the same error raised when processing the next corpus. How can I fix it?

[Huffon]

Hey, I encountered same issue. And just found it happened due to the change of PEP guide-line in Python 3.7. You'd better use Python 3.6 or look up the solution in references.

PEP 479 is enabled for all code in Python 3.7, meaning that StopIteration exceptions 
raised directly or indirectly in coroutines and generators are transformed into 
RuntimeError exceptions. (Contributed by Yury Selivanov in bpo-32670.)

[References] Solution: https://github.com/clips/pattern/issues/283#issuecomment-518561491 Reason: https://www.juandebravo.com/2018/11/09/python-37-stop-iteration/ Python 3.7 release note: https://docs.python.org/3/whatsnew/3.7.html#changes-in-python-behavior

[voidism]

@Huffon It works! Thank you!