Step6_zero_phrase_filtering problem

[Tamali6]

While training monoses I got an error in Step 7 which is------

Traceback (most recent call last): File "/home/xyz/monoses/training/tuning/tune.py", line 335, in main() File "/home/xyz/monoses/training/tuning/tune.py", line 322, in main extract_zmert_params(tmp + '/dcfg.txt.ZMERT.final')) File "/home/xyz/monoses/training/tuning/tune.py", line 73, in extract_zmert_params with open(path, encoding='utf-8', errors='surrogateescape') as f: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpv1m8y_i1/dcfg.txt.ZMERT.final' clean-corpus.perl: processing /home/xyz/models/monoses/src-tgt/tmpzbtcque6/train.bt & .trg to /home/xyz/models/monoses/src-tgt/tmpzbtcque6/train-supervised/clean, cutoff 3-80, ratio 9

From the log file and intermediate results, I find out that

• It successfully generated phrase tables.
• However, in step 6 it filtered 0% which I suspect.

P(f|e) filter limit: 100 Filtering using P(e|f) only. n=100

..................................................[n:500000] ..................................................[n:1000000] ..................................................[n:1500000] ..................................................[n:2000000] ..................................................[n:2500000] ..................................................[n:3000000] ..................................................[n:3500000] ..................................................[n:4000000] ..................................................[n:4500000] ..................................................[n:5000000] ..................................................[n:5500000] ..................................................[n:6000000] ..................................................[n:6500000] ..................................................[n:7000000] ..................................................[n:7500000] ..................................................[n:8000000] ..................................................[n:8500000] ..................................................[n:9000000] ..................................................[n:9500000] ..................................................[n:10000000]

unfiltered phrases pairs: 10000000

 P(f|e) filter [first]: 0   (0%)
   significance filter: 0   (0%)
        TOTAL FILTERED: 0   (0%)

FILTERED phrase pairs: 10000000   (100%)

• Then, in Step 7 while running decoder, it printed -

Call to decoder returned 1; was expecting 0. Z-MERT exiting prematurely (MertCore returned 30)...

[kellymarchisio]

I confirm that I have experienced this issue many times. The temporary directory is deleted before extract_zmert_params is called. The failure is intermittent

[kellymarchisio]

Ok, I finally figured out my related issue.

I got Z-MERT exiting prematurely (MertCore returned 1)...

This was due to moses2 segfaulting under the hood -> it segfaulted because one of the lines in the dev file I was passing into it was too long. I truncated each line in the dev set to 200 chars, and the segfault resolved. If you're doing unsupervised tuning, I recommend truncating the dev file you pass to moses2

Note: This also happened when I accidentally passed in two files for --supervised-tuning that were of different lengths.
Note2: Failure to use the moses tokenizer or escape-special-chars.perl script can also cause moses2 segfaults within zmert (https://github.com/moses-smt/mosesdecoder/tree/master/scripts/tokenizer)