pseudogen
pseudogen copied to clipboard
delihiros
Couldn't filter at /pseudogen/tools/travatar/script/mert/mert-travatar.pl line 90.
I get this error while building
Couldn't open travatar-model/model/travatar.ini
Exit code: 2
Couldn't filter at /pseudogen/tools/travatar/script/mert/mert-travatar.pl line 90.
The command '/bin/sh -c git clone https://github.com/delihiros/pseudogen.git && cd pseudogen && ./tool_setup.sh && mkdir data && cd data && wget -O- http://ahclab.naist.jp/pseudogen/en-django.tar.gz | tar zxvf - && mv en-django/all.* . && ../train-pseudogen.sh -p all.code -e all.anno' returned a non-zero code: 2
More logging following
tokenizing python ...
tokenizing english ...
parsing python ...
head insertion ...
simplifying ...
making data ...
making alignment ...
../train-pseudogen.sh: 52: ../train-pseudogen.sh: /pseudogen/tools/pialign/src/bin/pialign: not found
making language model ...
=== 1/5 Counting and sorting n-grams ===
Reading /pseudogen/data/train.entok
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
Unigram tokens 240407 types 6844
=== 2/5 Calculating and sorting adjusted counts ===
Chain sizes: 1:82128 2:980730240 3:1838869248 4:2942190592 5:4290694912
Statistics:
1 6844 D1=0.456042 D2=1.32836 D3+=1.95408
2 39124 D1=0.752714 D2=1.33396 D3+=1.60661
3 74152 D1=0.823118 D2=1.3625 D3+=1.55919
4 100578 D1=0.877647 D2=1.41071 D3+=1.59675
5 117034 D1=0.769353 D2=1.29688 D3+=1.40612
Memory estimate for binary LM:
type kB
probing 7243 assuming -p 1.5
probing 8523 assuming -r models -p 1.5
trie 3216 without quantization
trie 1668 assuming -q 8 -b 8 quantization
trie 2965 assuming -a 22 array pointer compression
trie 1416 assuming -a 22 -q 8 -b 8 array pointer compression and quantization
=== 3/5 Calculating and sorting initial probabilities ===
Chain sizes: 1:82128 2:625984 3:1483040 4:2413872 5:3276952
=== 4/5 Calculating and writing order-interpolated probabilities ===
Chain sizes: 1:82128 2:625984 3:1483040 4:2413872 5:3276952
Chain sizes: 1:82128 2:625984 3:1483040 4:2413872 5:3276952
=== 5/5 Writing ARPA model ===
Name:lt-lmplz VmPeak:10002084 kB VmRSS:8088 kB RSSMax:1755116 kB user:0.779529 sys:0.834626 CPU:1.61416 real:1.41535
Reading lm/lm.arpa
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
SUCCESS
training travatar ...
Executing: mkdir travatar-model
(1) Preparing data @ Mon May 11 10:32:46 UTC 2020
Executing: mkdir -p travatar-model/data
Executing: /pseudogen/tools/travatar/src/bin/tree-converter -input_format penn -output_format word < train.reducedtree > travatar-model/data/src.word
Main arguments:
Optional arguments:
-input_format penn
-output_format word
-split
-compoundsplit
-compoundsplit_filler
-compoundsplit_threshold 0.01
-compoundsplit_minchar 3
-binarize none
-case none
-flatten false
-debug 0
Transforming trees (.=10,000, !=100,000 sentences)
.
(2) Creating alignments @ Mon May 11 10:32:47 UTC 2020
Executing: mkdir -p travatar-model/align
Executing: /pseudogen/tools/giza-pp/mkcls -c50 -n2 -ptrain.entok -Vtravatar-model/align/trg.vcb.classes opt
Executing: /pseudogen/tools/giza-pp/mkcls -c50 -n2 -ptravatar-model/data/src.word -Vtravatar-model/align/src.vcb.classes opt
***** 2 runs. (algorithm:TA)*****
;KategProblem:cats: 50 words: 7026
start-costs: MEAN: 1.57369e+06 (1.57369e+06-1.57369e+06) SIGMA:0.702256
end-costs: MEAN: 1.44237e+06 (1.44218e+06-1.44255e+06) SIGMA:185.301
start-pp: MEAN: 99.2817 (99.2812-99.2822) SIGMA:0.000498938
end-pp: MEAN: 38.8095 (38.7581-38.8609) SIGMA:0.0514388
iterations: MEAN: 192496 (191162-193831) SIGMA:1334.5
time: MEAN: 3.09004 (3.08413-3.09595) SIGMA:0.005906
***** 2 runs. (algorithm:TA)*****
;KategProblem:cats: 50 words: 6842
start-costs: MEAN: 3.0405e+06 (3.03629e+06-3.04472e+06) SIGMA:4217.96
end-costs: MEAN: 2.77351e+06 (2.77251e+06-2.77451e+06) SIGMA:995.689
start-pp: MEAN: 90.8114 (89.3224-92.3005) SIGMA:1.48907
end-pp: MEAN: 32.1567 (32.0323-32.2812) SIGMA:0.124481
iterations: MEAN: 204423 (202255-206591) SIGMA:2168
time: MEAN: 4.4065 (4.36276-4.45023) SIGMA:0.0437335
Executing: /pseudogen/tools/giza-pp/snt2cooc.out travatar-model/align/src.vcb travatar-model/align/trg.vcb travatar-model/align/src-trg.snt > travatar-model/align/src-trg.cooc
line 1000
line 2000
line 3000
line 4000
line 5000
line 6000
line 7000
line 8000
line 9000
line 10000
line 11000
line 12000
line 13000
line 14000
line 15000
line 16000
END.
Executing: /pseudogen/tools/giza-pp/snt2cooc.out travatar-model/align/trg.vcb travatar-model/align/src.vcb travatar-model/align/trg-src.snt > travatar-model/align/trg-src.cooc
line 1000
line 2000
line 3000
line 4000
line 5000
line 6000
line 7000
line 8000
line 9000
line 10000
line 11000
line 12000
line 13000
line 14000
line 15000
line 16000
END.
Executing: /pseudogen/tools/giza-pp/GIZA++ -CoocurrenceFile travatar-model/align/trg-src.cooc -c travatar-model/align/trg-src.snt -m1 5 -m2 0 -m3 3 -m4 3 -model1dumpfrequency 1 -model4smoothfactor 0.4 -nodumps 1 -nsmooth 4 -o travatar-model/align/trg-src.giza -onlyaldumps 1 -p0 0.999 -s travatar-model/align/trg.vcb -t travatar-model/align/src.vcb
Executing: /pseudogen/tools/giza-pp/GIZA++ -CoocurrenceFile travatar-model/align/src-trg.cooc -c travatar-model/align/src-trg.snt -m1 5 -m2 0 -m3 3 -m4 3 -model1dumpfrequency 1 -model4smoothfactor 0.4 -nodumps 1 -nsmooth 4 -o travatar-model/align/src-trg.giza -onlyaldumps 1 -p0 0.999 -s travatar-model/align/src.vcb -t travatar-model/align/trg.vcb
ERROR: Execution of: /pseudogen/tools/giza-pp/GIZA++ -CoocurrenceFile travatar-model/align/src-trg.cooc -c travatar-model/align/src-trg.snt -m1 5 -m2 0 -m3 3 -m4 3 -model1dumpfrequency 1 -model4smoothfactor 0.4 -nodumps 1 -nsmooth 4 -o travatar-model/align/src-trg.giza -onlyaldumps 1 -p0 0.999 -s travatar-model/align/src.vcb -t travatar-model/align/trg.vcb
died with signal 11, without coredump
ERROR: Execution of: /pseudogen/tools/giza-pp/GIZA++ -CoocurrenceFile travatar-model/align/trg-src.cooc -c travatar-model/align/trg-src.snt -m1 5 -m2 0 -m3 3 -m4 3 -model1dumpfrequency 1 -model4smoothfactor 0.4 -nodumps 1 -nsmooth 4 -o travatar-model/align/trg-src.giza -onlyaldumps 1 -p0 0.999 -s travatar-model/align/trg.vcb -t travatar-model/align/src.vcb
died with signal 11, without coredump
tuning travatar ...
Executing: mkdir tune
Executing: /pseudogen/tools/travatar/script/train/filter-model.pl travatar-model/model/travatar.ini tune/run1.ini tune/filtered "/pseudogen/tools/travatar/script/train/filter-rule-table.py dev.reducedtree"
Couldn't open travatar-model/model/travatar.ini
Exit code: 2
Couldn't filter at /pseudogen/tools/travatar/script/mert/mert-travatar.pl line 90.
The command '/bin/sh -c git clone https://github.com/delihiros/pseudogen.git && cd pseudogen && ./tool_setup.sh && mkdir data && cd data && wget -O- http://ahclab.naist.jp/pseudogen/en-django.tar.gz | tar zxvf - && mv en-django/all.* . && ../train-pseudogen.sh -p all.code -e all.anno' returned a non-zero code: 2
I have a similar problem. My execution also can't open travatar-model/model/travatar.ini.
UPD: Apparently, Couldn't open travatar-model/model/travatar.ini happens when the previous step fails. I had a similar problem and before the error I had another one. After I fixed the original issue, it all worked fine.
E.g. in the issue description another error happens before it fails to open travatar.ini:
ERROR: Execution of: /pseudogen/tools/giza-pp/GIZA++
@postatum can you describe how you fixed this?
I don't exactly remember how I fixed mine, but from I remember, the error Couldn't open travatar-model/model/travatar.ini is caused by other errors happening before it. Address those and opening travatar.ini should succeed. And how you address previous errors depends on what those error logs say. I think googling errors logs should suggest solutions.
Hope this helps.