Opus-MT icon indicating copy to clipboard operation
Opus-MT copied to clipboard

Published 20 hours ago • verified publisher icon Helsinki-NLP

Docker compile for CPU fails due to sentencepiece.pc missing

Open akangasr opened this issue 4 years ago • 9 comments

When compiling the dockerfile with docker build . error is raised at make install step of marian compilation:

CMake Error at src/3rd_party/sentencepiece/cmake_install.cmake:41 (file):
  file INSTALL cannot find "/usr/src/app/marian/sentencepiece.pc".
Call Stack (most recent call first):
  src/3rd_party/cmake_install.cmake:46 (include)
  src/cmake_install.cmake:42 (include)
  cmake_install.cmake:42 (include)

akangasr avatar Jun 18 '20 14:06 akangasr

I have created a fix for this: https://github.com/Helsinki-NLP/Opus-MT/pull/22 Can you please verify if that resolves the issue?

This was a regression with update to MarianNMT 1.9

santhoshtr avatar Jun 19 '20 11:06 santhoshtr

Had the same error. With new fix the image builds but when running:

opus-mt_1 | marian-server: error while loading shared libraries: libsentencepiece_train.so.0: cannot open shared object file: No such file or directory opus-mt_1 | ERROR:asyncio:Future exception was never retrieved opus-mt_1 | future: <Future finished exception=CalledProcessError(127, 'unknown')> opus-mt_1 | Traceback (most recent call last): opus-mt_1 | File "/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 742, in run opus-mt_1 | yielded = self.gen.throw(*exc_info) # type: ignore opus-mt_1 | File "server.py", line 49, in run opus-mt_1 | ret = yield self.p.wait_for_exit() opus-mt_1 | File "/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 735, in run opus-mt_1 | value = future.result() opus-mt_1 | subprocess.CalledProcessError: Command 'unknown' returned non-zero exit status 127.

y-bo avatar Jun 22 '20 13:06 y-bo

That is strange. I am sharing the output of docker run opus-mt log below. Can you find where does it differs from your result?

docker run opus-mt log 

docker run opus-mt
[2020-06-23 03:56:34] [marian] Marian v1.9.0 3c7a88f4 2020-03-10 11:34:07 -0700
[2020-06-23 03:56:34] [marian] Running on 700b5d8fff2d as process 11 with command line:
[2020-06-23 03:56:34] [marian] marian-server -c ./models/en-es/decoder.yml -p 10001 --allow-unk -b 6 --mini-batch 64 --normalize 0.6 --maxi-batch-sort src --maxi-batch 100
[2020-06-23 03:56:34] [config] alignment: ""
[2020-06-23 03:56:34] [config] allow-unk: true
[2020-06-23 03:56:34] [config] authors: false
[2020-06-23 03:56:34] [config] beam-size: 6
[2020-06-23 03:56:34] [config] bert-class-symbol: "[CLS]"
[2020-06-23 03:56:34] [config] bert-mask-symbol: "[MASK]"
[2020-06-23 03:56:34] [config] bert-masking-fraction: 0.15
[2020-06-23 03:56:34] [config] bert-sep-symbol: "[SEP]"
[2020-06-23 03:56:34] [config] bert-train-type-embeddings: true
[2020-06-23 03:56:34] [config] bert-type-vocab-size: 2
[2020-06-23 03:56:34] [config] best-deep: false
[2020-06-23 03:56:34] [config] build-info: ""
[2020-06-23 03:56:34] [config] cite: false
[2020-06-23 03:56:34] [config] clip-gemm: 0
[2020-06-23 03:56:34] [config] cpu-threads: 1
[2020-06-23 03:56:34] [config] dec-cell: gru
[2020-06-23 03:56:34] [config] dec-cell-base-depth: 2
[2020-06-23 03:56:34] [config] dec-cell-high-depth: 1
[2020-06-23 03:56:34] [config] dec-depth: 6
[2020-06-23 03:56:34] [config] devices:
[2020-06-23 03:56:34] [config]   - 0
[2020-06-23 03:56:34] [config] dim-emb: 512
[2020-06-23 03:56:34] [config] dim-rnn: 1024
[2020-06-23 03:56:34] [config] dim-vocabs:
[2020-06-23 03:56:34] [config]   - 65000
[2020-06-23 03:56:34] [config]   - 65000
[2020-06-23 03:56:34] [config] dump-config: ""
[2020-06-23 03:56:34] [config] enc-cell: gru
[2020-06-23 03:56:34] [config] enc-cell-depth: 1
[2020-06-23 03:56:34] [config] enc-depth: 6
[2020-06-23 03:56:34] [config] enc-type: bidirectional
[2020-06-23 03:56:34] [config] ignore-model-config: false
[2020-06-23 03:56:34] [config] input:
[2020-06-23 03:56:34] [config]   - stdin
[2020-06-23 03:56:34] [config] input-types:
[2020-06-23 03:56:34] [config]   []
[2020-06-23 03:56:34] [config] interpolate-env-vars: false
[2020-06-23 03:56:34] [config] layer-normalization: false
[2020-06-23 03:56:34] [config] lemma-dim-emb: 0
[2020-06-23 03:56:34] [config] log: ""
[2020-06-23 03:56:34] [config] log-level: info
[2020-06-23 03:56:34] [config] log-time-zone: ""
[2020-06-23 03:56:34] [config] max-length: 1000
[2020-06-23 03:56:34] [config] max-length-crop: false
[2020-06-23 03:56:34] [config] max-length-factor: 3
[2020-06-23 03:56:34] [config] maxi-batch: 100
[2020-06-23 03:56:34] [config] maxi-batch-sort: src
[2020-06-23 03:56:34] [config] mini-batch: 64
[2020-06-23 03:56:34] [config] mini-batch-words: 0
[2020-06-23 03:56:34] [config] models:
[2020-06-23 03:56:34] [config]   - /usr/src/app/models/en-es/opus.bpe32k-bpe32k.transformer.model1.npz.best-perplexity.npz
[2020-06-23 03:56:34] [config] n-best: false
[2020-06-23 03:56:34] [config] no-spm-decode: false
[2020-06-23 03:56:34] [config] normalize: 0.6
[2020-06-23 03:56:34] [config] num-devices: 0
[2020-06-23 03:56:34] [config] optimize: false
[2020-06-23 03:56:34] [config] output: stdout
[2020-06-23 03:56:34] [config] output-sampling: false
[2020-06-23 03:56:34] [config] port: 10001
[2020-06-23 03:56:34] [config] precision:
[2020-06-23 03:56:34] [config]   - float32
[2020-06-23 03:56:34] [config] quiet: false
[2020-06-23 03:56:34] [config] quiet-translation: false
[2020-06-23 03:56:34] [config] relative-paths: false
[2020-06-23 03:56:34] [config] right-left: false
[2020-06-23 03:56:34] [config] seed: 0
[2020-06-23 03:56:34] [config] shortlist:
[2020-06-23 03:56:34] [config]   []
[2020-06-23 03:56:34] [config] skip: false
[2020-06-23 03:56:34] [config] skip-cost: false
[2020-06-23 03:56:34] [config] tied-embeddings: false
[2020-06-23 03:56:34] [config] tied-embeddings-all: true
[2020-06-23 03:56:34] [config] tied-embeddings-src: false
[2020-06-23 03:56:34] [config] transformer-aan-activation: swish
[2020-06-23 03:56:34] [config] transformer-aan-depth: 2
[2020-06-23 03:56:34] [config] transformer-aan-nogate: false
[2020-06-23 03:56:34] [config] transformer-decoder-autoreg: self-attention
[2020-06-23 03:56:34] [config] transformer-depth-scaling: false
[2020-06-23 03:56:34] [config] transformer-dim-aan: 2048
[2020-06-23 03:56:34] [config] transformer-dim-ffn: 2048
[2020-06-23 03:56:34] [config] transformer-ffn-activation: swish
[2020-06-23 03:56:34] [config] transformer-ffn-depth: 2
[2020-06-23 03:56:34] [config] transformer-guided-alignment-layer: last
[2020-06-23 03:56:34] [config] transformer-heads: 8
[2020-06-23 03:56:34] [config] transformer-no-projection: false
[2020-06-23 03:56:34] [config] transformer-postprocess: dan
[2020-06-23 03:56:34] [config] transformer-postprocess-emb: d
[2020-06-23 03:56:34] [config] transformer-preprocess: ""
[2020-06-23 03:56:34] [config] transformer-tied-layers:
[2020-06-23 03:56:34] [config]   []
[2020-06-23 03:56:34] [config] transformer-train-position-embeddings: false
[2020-06-23 03:56:34] [config] type: transformer
[2020-06-23 03:56:34] [config] version: v1.7.6 1d4ba73 2019-05-11 17:16:31 +0100
[2020-06-23 03:56:34] [config] vocabs:
[2020-06-23 03:56:34] [config]   - /usr/src/app/models/en-es/opus.bpe32k-bpe32k.vocab.yml
[2020-06-23 03:56:34] [config]   - /usr/src/app/models/en-es/opus.bpe32k-bpe32k.vocab.yml
[2020-06-23 03:56:34] [config] weights:
[2020-06-23 03:56:34] [config]   []
[2020-06-23 03:56:34] [config] word-penalty: 0
[2020-06-23 03:56:34] [config] word-scores: false
[2020-06-23 03:56:34] [config] workspace: 512
[2020-06-23 03:56:34] [config] Loaded model has been created with Marian v1.7.6 1d4ba73 2019-05-11 17:16:31 +0100
[2020-06-23 03:56:34] [data] Loading vocabulary from JSON/Yaml file /usr/src/app/models/en-es/opus.bpe32k-bpe32k.vocab.yml
[2020-06-23 03:56:35] [marian] Marian v1.9.0 3c7a88f4 2020-03-10 11:34:07 -0700
[2020-06-23 03:56:35] [marian] Running on 700b5d8fff2d as process 14 with command line:
[2020-06-23 03:56:35] [marian] marian-server -c ./models/en-fi/decoder.yml -p 10002 --allow-unk -b 6 --mini-batch 64 --normalize 0.6 --maxi-batch-sort src --maxi-batch 100
[2020-06-23 03:56:35] [config] alignment: ""
[2020-06-23 03:56:35] [config] allow-unk: true
[2020-06-23 03:56:35] [config] authors: false
[2020-06-23 03:56:35] [config] beam-size: 6
[2020-06-23 03:56:35] [config] bert-class-symbol: "[CLS]"
[2020-06-23 03:56:35] [config] bert-mask-symbol: "[MASK]"
[2020-06-23 03:56:35] [config] bert-masking-fraction: 0.15
[2020-06-23 03:56:35] [config] bert-sep-symbol: "[SEP]"
[2020-06-23 03:56:35] [config] bert-train-type-embeddings: true
[2020-06-23 03:56:35] [config] bert-type-vocab-size: 2
[2020-06-23 03:56:35] [config] best-deep: false
[2020-06-23 03:56:35] [config] build-info: ""
[2020-06-23 03:56:35] [config] cite: false
[2020-06-23 03:56:35] [config] clip-gemm: 0
[2020-06-23 03:56:35] [config] cpu-threads: 1
[2020-06-23 03:56:35] [config] dec-cell: gru
[2020-06-23 03:56:35] [config] dec-cell-base-depth: 2
[2020-06-23 03:56:35] [config] dec-cell-high-depth: 1
[2020-06-23 03:56:35] [config] dec-depth: 6
[2020-06-23 03:56:35] [config] devices:
[2020-06-23 03:56:35] [config]   - 0
[2020-06-23 03:56:35] [config] dim-emb: 512
[2020-06-23 03:56:35] [config] dim-rnn: 1024
[2020-06-23 03:56:35] [config] dim-vocabs:
[2020-06-23 03:56:35] [config]   - 61547
[2020-06-23 03:56:35] [config]   - 61547
[2020-06-23 03:56:35] [config] dump-config: ""
[2020-06-23 03:56:35] [config] enc-cell: gru
[2020-06-23 03:56:35] [config] enc-cell-depth: 1
[2020-06-23 03:56:35] [config] enc-depth: 6
[2020-06-23 03:56:35] [config] enc-type: bidirectional
[2020-06-23 03:56:35] [config] ignore-model-config: false
[2020-06-23 03:56:35] [config] input:
[2020-06-23 03:56:35] [config]   - stdin
[2020-06-23 03:56:35] [config] input-types:
[2020-06-23 03:56:35] [config]   []
[2020-06-23 03:56:35] [config] interpolate-env-vars: false
[2020-06-23 03:56:35] [config] layer-normalization: false
[2020-06-23 03:56:35] [config] lemma-dim-emb: 0
[2020-06-23 03:56:35] [config] log: ""
[2020-06-23 03:56:35] [config] log-level: info
[2020-06-23 03:56:35] [config] log-time-zone: ""
[2020-06-23 03:56:35] [config] max-length: 1000
[2020-06-23 03:56:35] [config] max-length-crop: false
[2020-06-23 03:56:35] [config] max-length-factor: 3
[2020-06-23 03:56:35] [config] maxi-batch: 100
[2020-06-23 03:56:35] [config] maxi-batch-sort: src
[2020-06-23 03:56:35] [config] mini-batch: 64
[2020-06-23 03:56:35] [config] mini-batch-words: 0
[2020-06-23 03:56:35] [config] models:
[2020-06-23 03:56:35] [config]   - /usr/src/app/models/en-fi/opus.spm32k-spm32k.transformer-align.model1.npz.best-perplexity.npz
[2020-06-23 03:56:35] [config] n-best: false
[2020-06-23 03:56:35] [config] no-spm-decode: false
[2020-06-23 03:56:35] [config] normalize: 0.6
[2020-06-23 03:56:35] [config] num-devices: 0
[2020-06-23 03:56:35] [config] optimize: false
[2020-06-23 03:56:35] [config] output: stdout
[2020-06-23 03:56:35] [config] output-sampling: false
[2020-06-23 03:56:35] [config] port: 10002
[2020-06-23 03:56:35] [config] precision:
[2020-06-23 03:56:35] [config]   - float32
[2020-06-23 03:56:35] [config] quiet: false
[2020-06-23 03:56:35] [config] quiet-translation: false
[2020-06-23 03:56:35] [config] relative-paths: false
[2020-06-23 03:56:35] [config] right-left: false
[2020-06-23 03:56:35] [config] seed: 0
[2020-06-23 03:56:35] [config] shortlist:
[2020-06-23 03:56:35] [config]   []
[2020-06-23 03:56:35] [config] skip: false
[2020-06-23 03:56:35] [config] skip-cost: false
[2020-06-23 03:56:35] [config] tied-embeddings: false
[2020-06-23 03:56:35] [config] tied-embeddings-all: true
[2020-06-23 03:56:35] [config] tied-embeddings-src: false
[2020-06-23 03:56:35] [config] transformer-aan-activation: swish
[2020-06-23 03:56:35] [config] transformer-aan-depth: 2
[2020-06-23 03:56:35] [config] transformer-aan-nogate: false
[2020-06-23 03:56:35] [config] transformer-decoder-autoreg: self-attention
[2020-06-23 03:56:35] [config] transformer-depth-scaling: false
[2020-06-23 03:56:35] [config] transformer-dim-aan: 2048
[2020-06-23 03:56:35] [config] transformer-dim-ffn: 2048
[2020-06-23 03:56:35] [config] transformer-ffn-activation: swish
[2020-06-23 03:56:35] [config] transformer-ffn-depth: 2
[2020-06-23 03:56:35] [config] transformer-guided-alignment-layer: last
[2020-06-23 03:56:35] [config] transformer-heads: 8
[2020-06-23 03:56:35] [config] transformer-no-projection: false
[2020-06-23 03:56:35] [config] transformer-postprocess: dan
[2020-06-23 03:56:35] [config] transformer-postprocess-emb: d
[2020-06-23 03:56:35] [config] transformer-preprocess: ""
[2020-06-23 03:56:35] [config] transformer-tied-layers:
[2020-06-23 03:56:35] [config]   []
[2020-06-23 03:56:35] [config] transformer-train-position-embeddings: false
[2020-06-23 03:56:35] [config] type: transformer
[2020-06-23 03:56:35] [config] ulr: false
[2020-06-23 03:56:35] [config] ulr-dim-emb: 0
[2020-06-23 03:56:35] [config] ulr-trainable-transformation: false
[2020-06-23 03:56:35] [config] version: v1.8.2 2111c28 2019-10-16 08:36:48 -0700
[2020-06-23 03:56:35] [config] vocabs:
[2020-06-23 03:56:35] [config]   - /usr/src/app/models/en-fi/opus.spm32k-spm32k.vocab.yml
[2020-06-23 03:56:35] [config]   - /usr/src/app/models/en-fi/opus.spm32k-spm32k.vocab.yml
[2020-06-23 03:56:35] [config] weights:
[2020-06-23 03:56:35] [config]   []
[2020-06-23 03:56:35] [config] word-penalty: 0
[2020-06-23 03:56:35] [config] word-scores: false
[2020-06-23 03:56:35] [config] workspace: 512
[2020-06-23 03:56:35] [config] Loaded model has been created with Marian v1.8.2 2111c28 2019-10-16 08:36:48 -0700
[2020-06-23 03:56:35] [data] Loading vocabulary from JSON/Yaml file /usr/src/app/models/en-fi/opus.spm32k-spm32k.vocab.yml
[2020-06-23 03:56:35] [marian] Marian v1.9.0 3c7a88f4 2020-03-10 11:34:07 -0700
[2020-06-23 03:56:35] [marian] Running on 700b5d8fff2d as process 17 with command line:
[2020-06-23 03:56:35] [marian] marian-server -c ./models/en-ml/decoder.yml -p 10003 --allow-unk -b 6 --mini-batch 64 --normalize 0.6 --maxi-batch-sort src --maxi-batch 100
[2020-06-23 03:56:35] [config] alignment: ""
[2020-06-23 03:56:35] [config] allow-unk: true
[2020-06-23 03:56:35] [config] authors: false
[2020-06-23 03:56:35] [config] beam-size: 6
[2020-06-23 03:56:35] [config] bert-class-symbol: "[CLS]"
[2020-06-23 03:56:35] [config] bert-mask-symbol: "[MASK]"
[2020-06-23 03:56:35] [config] bert-masking-fraction: 0.15
[2020-06-23 03:56:35] [config] bert-sep-symbol: "[SEP]"
[2020-06-23 03:56:35] [config] bert-train-type-embeddings: true
[2020-06-23 03:56:35] [config] bert-type-vocab-size: 2
[2020-06-23 03:56:35] [config] best-deep: false
[2020-06-23 03:56:35] [config] build-info: ""
[2020-06-23 03:56:35] [config] cite: false
[2020-06-23 03:56:35] [config] clip-gemm: 0
[2020-06-23 03:56:35] [config] cpu-threads: 1
[2020-06-23 03:56:35] [config] dec-cell: gru
[2020-06-23 03:56:35] [config] dec-cell-base-depth: 2
[2020-06-23 03:56:35] [config] dec-cell-high-depth: 1
[2020-06-23 03:56:35] [config] dec-depth: 6
[2020-06-23 03:56:35] [config] devices:
[2020-06-23 03:56:35] [config]   - 0
[2020-06-23 03:56:35] [config] dim-emb: 512
[2020-06-23 03:56:35] [config] dim-rnn: 1024
[2020-06-23 03:56:35] [config] dim-vocabs:
[2020-06-23 03:56:35] [config]   - 63223
[2020-06-23 03:56:35] [config]   - 63223
[2020-06-23 03:56:35] [config] dump-config: ""
[2020-06-23 03:56:35] [config] enc-cell: gru
[2020-06-23 03:56:35] [config] enc-cell-depth: 1
[2020-06-23 03:56:35] [config] enc-depth: 6
[2020-06-23 03:56:35] [config] enc-type: bidirectional
[2020-06-23 03:56:35] [config] ignore-model-config: false
[2020-06-23 03:56:35] [config] input:
[2020-06-23 03:56:35] [config]   - stdin
[2020-06-23 03:56:35] [config] input-types:
[2020-06-23 03:56:35] [config]   []
[2020-06-23 03:56:35] [config] interpolate-env-vars: false
[2020-06-23 03:56:35] [config] layer-normalization: false
[2020-06-23 03:56:35] [config] lemma-dim-emb: 0
[2020-06-23 03:56:35] [config] log: ""
[2020-06-23 03:56:35] [config] log-level: info
[2020-06-23 03:56:35] [config] log-time-zone: ""
[2020-06-23 03:56:35] [config] max-length: 1000
[2020-06-23 03:56:35] [config] max-length-crop: false
[2020-06-23 03:56:35] [config] max-length-factor: 3
[2020-06-23 03:56:35] [config] maxi-batch: 100
[2020-06-23 03:56:35] [config] maxi-batch-sort: src
[2020-06-23 03:56:35] [config] mini-batch: 64
[2020-06-23 03:56:35] [config] mini-batch-words: 0
[2020-06-23 03:56:35] [config] models:
[2020-06-23 03:56:35] [config]   - /usr/src/app/models/en-ml/opus+bt.spm32k-spm32k.transformer-align.model1.npz.best-perplexity.npz
[2020-06-23 03:56:35] [config] n-best: false
[2020-06-23 03:56:35] [config] no-spm-decode: false
[2020-06-23 03:56:35] [config] normalize: 0.6
[2020-06-23 03:56:35] [config] num-devices: 0
[2020-06-23 03:56:35] [config] optimize: false
[2020-06-23 03:56:35] [config] output: stdout
[2020-06-23 03:56:35] [config] output-sampling: false
[2020-06-23 03:56:35] [config] port: 10003
[2020-06-23 03:56:35] [config] precision:
[2020-06-23 03:56:35] [config]   - float32
[2020-06-23 03:56:35] [config] quiet: false
[2020-06-23 03:56:35] [config] quiet-translation: false
[2020-06-23 03:56:35] [config] relative-paths: false
[2020-06-23 03:56:35] [config] right-left: false
[2020-06-23 03:56:35] [config] seed: 0
[2020-06-23 03:56:35] [config] shortlist:
[2020-06-23 03:56:35] [config]   []
[2020-06-23 03:56:35] [config] skip: false
[2020-06-23 03:56:35] [config] skip-cost: false
[2020-06-23 03:56:35] [config] tied-embeddings: false
[2020-06-23 03:56:35] [config] tied-embeddings-all: true
[2020-06-23 03:56:35] [config] tied-embeddings-src: false
[2020-06-23 03:56:35] [config] transformer-aan-activation: swish
[2020-06-23 03:56:35] [config] transformer-aan-depth: 2
[2020-06-23 03:56:35] [config] transformer-aan-nogate: false
[2020-06-23 03:56:35] [config] transformer-decoder-autoreg: self-attention
[2020-06-23 03:56:35] [config] transformer-depth-scaling: false
[2020-06-23 03:56:35] [config] transformer-dim-aan: 2048
[2020-06-23 03:56:35] [config] transformer-dim-ffn: 2048
[2020-06-23 03:56:35] [config] transformer-ffn-activation: swish
[2020-06-23 03:56:35] [config] transformer-ffn-depth: 2
[2020-06-23 03:56:35] [config] transformer-guided-alignment-layer: last
[2020-06-23 03:56:35] [config] transformer-heads: 8
[2020-06-23 03:56:35] [config] transformer-no-projection: false
[2020-06-23 03:56:35] [config] transformer-postprocess: dan
[2020-06-23 03:56:35] [config] transformer-postprocess-emb: d
[2020-06-23 03:56:35] [config] transformer-preprocess: ""
[2020-06-23 03:56:35] [config] transformer-tied-layers:
[2020-06-23 03:56:35] [config]   []
[2020-06-23 03:56:35] [config] transformer-train-position-embeddings: false
[2020-06-23 03:56:35] [config] type: transformer
[2020-06-23 03:56:35] [config] ulr: false
[2020-06-23 03:56:35] [config] ulr-dim-emb: 0
[2020-06-23 03:56:35] [config] ulr-trainable-transformation: false
[2020-06-23 03:56:35] [config] version: v1.8.2 2111c28 2019-10-16 08:36:48 -0700
[2020-06-23 03:56:35] [config] vocabs:
[2020-06-23 03:56:35] [config]   - /usr/src/app/models/en-ml/opus+bt.spm32k-spm32k.vocab.yml
[2020-06-23 03:56:35] [config]   - /usr/src/app/models/en-ml/opus+bt.spm32k-spm32k.vocab.yml
[2020-06-23 03:56:35] [config] weights:
[2020-06-23 03:56:35] [config]   []
[2020-06-23 03:56:35] [config] word-penalty: 0
[2020-06-23 03:56:35] [config] word-scores: false
[2020-06-23 03:56:35] [config] workspace: 512
[2020-06-23 03:56:35] [config] Loaded model has been created with Marian v1.8.2 2111c28 2019-10-16 08:36:48 -0700
[2020-06-23 03:56:35] [data] Loading vocabulary from JSON/Yaml file /usr/src/app/models/en-ml/opus+bt.spm32k-spm32k.vocab.yml
[2020-06-23 03:56:35] [marian] Marian v1.9.0 3c7a88f4 2020-03-10 11:34:07 -0700
[2020-06-23 03:56:35] [marian] Running on 700b5d8fff2d as process 20 with command line:
[2020-06-23 03:56:35] [marian] marian-server -c ./models/en-mr/decoder.yml -p 10004 --allow-unk -b 6 --mini-batch 64 --normalize 0.6 --maxi-batch-sort src --maxi-batch 100
[2020-06-23 03:56:35] [config] alignment: ""
[2020-06-23 03:56:35] [config] allow-unk: true
[2020-06-23 03:56:35] [config] authors: false
[2020-06-23 03:56:35] [config] beam-size: 6
[2020-06-23 03:56:35] [config] bert-class-symbol: "[CLS]"
[2020-06-23 03:56:35] [config] bert-mask-symbol: "[MASK]"
[2020-06-23 03:56:35] [config] bert-masking-fraction: 0.15
[2020-06-23 03:56:35] [config] bert-sep-symbol: "[SEP]"
[2020-06-23 03:56:35] [config] bert-train-type-embeddings: true
[2020-06-23 03:56:35] [config] bert-type-vocab-size: 2
[2020-06-23 03:56:35] [config] best-deep: false
[2020-06-23 03:56:35] [config] build-info: ""
[2020-06-23 03:56:35] [config] cite: false
[2020-06-23 03:56:35] [config] clip-gemm: 0
[2020-06-23 03:56:35] [config] cpu-threads: 1
[2020-06-23 03:56:35] [config] dec-cell: gru
[2020-06-23 03:56:35] [config] dec-cell-base-depth: 2
[2020-06-23 03:56:35] [config] dec-cell-high-depth: 1
[2020-06-23 03:56:35] [config] dec-depth: 6
[2020-06-23 03:56:35] [config] devices:
[2020-06-23 03:56:35] [config]   - 0
[2020-06-23 03:56:35] [config] dim-emb: 512
[2020-06-23 03:56:35] [config] dim-rnn: 1024
[2020-06-23 03:56:35] [config] dim-vocabs:
[2020-06-23 03:56:35] [config]   - 61673
[2020-06-23 03:56:35] [config]   - 61673
[2020-06-23 03:56:35] [config] dump-config: ""
[2020-06-23 03:56:35] [config] enc-cell: gru
[2020-06-23 03:56:35] [config] enc-cell-depth: 1
[2020-06-23 03:56:35] [config] enc-depth: 6
[2020-06-23 03:56:35] [config] enc-type: bidirectional
[2020-06-23 03:56:35] [config] ignore-model-config: false
[2020-06-23 03:56:35] [config] input:
[2020-06-23 03:56:35] [config]   - stdin
[2020-06-23 03:56:35] [config] input-types:
[2020-06-23 03:56:35] [config]   []
[2020-06-23 03:56:35] [config] interpolate-env-vars: false
[2020-06-23 03:56:35] [config] layer-normalization: false
[2020-06-23 03:56:35] [config] lemma-dim-emb: 0
[2020-06-23 03:56:35] [config] log: ""
[2020-06-23 03:56:35] [config] log-level: info
[2020-06-23 03:56:35] [config] log-time-zone: ""
[2020-06-23 03:56:35] [config] max-length: 1000
[2020-06-23 03:56:35] [config] max-length-crop: false
[2020-06-23 03:56:35] [config] max-length-factor: 3
[2020-06-23 03:56:35] [config] maxi-batch: 100
[2020-06-23 03:56:35] [config] maxi-batch-sort: src
[2020-06-23 03:56:35] [config] mini-batch: 64
[2020-06-23 03:56:35] [config] mini-batch-words: 0
[2020-06-23 03:56:35] [config] models:
[2020-06-23 03:56:35] [config]   - /usr/src/app/models/en-mr/opus.spm32k-spm32k.transformer-align.model1.npz.best-perplexity.npz
[2020-06-23 03:56:35] [config] n-best: false
[2020-06-23 03:56:35] [config] no-spm-decode: false
[2020-06-23 03:56:35] [config] normalize: 0.6
[2020-06-23 03:56:35] [config] num-devices: 0
[2020-06-23 03:56:35] [config] optimize: false
[2020-06-23 03:56:35] [config] output: stdout
[2020-06-23 03:56:35] [config] output-sampling: false
[2020-06-23 03:56:35] [config] port: 10004
[2020-06-23 03:56:35] [config] precision:
[2020-06-23 03:56:35] [config]   - float32
[2020-06-23 03:56:35] [config] quiet: false
[2020-06-23 03:56:35] [config] quiet-translation: false
[2020-06-23 03:56:35] [config] relative-paths: false
[2020-06-23 03:56:35] [config] right-left: false
[2020-06-23 03:56:35] [config] seed: 0
[2020-06-23 03:56:35] [config] shortlist:
[2020-06-23 03:56:35] [config]   []
[2020-06-23 03:56:35] [config] skip: false
[2020-06-23 03:56:35] [config] skip-cost: false
[2020-06-23 03:56:35] [config] tied-embeddings: false
[2020-06-23 03:56:35] [config] tied-embeddings-all: true
[2020-06-23 03:56:35] [config] tied-embeddings-src: false
[2020-06-23 03:56:35] [config] transformer-aan-activation: swish
[2020-06-23 03:56:35] [config] transformer-aan-depth: 2
[2020-06-23 03:56:35] [config] transformer-aan-nogate: false
[2020-06-23 03:56:35] [config] transformer-decoder-autoreg: self-attention
[2020-06-23 03:56:35] [config] transformer-depth-scaling: false
[2020-06-23 03:56:35] [config] transformer-dim-aan: 2048
[2020-06-23 03:56:35] [config] transformer-dim-ffn: 2048
[2020-06-23 03:56:35] [config] transformer-ffn-activation: swish
[2020-06-23 03:56:35] [config] transformer-ffn-depth: 2
[2020-06-23 03:56:35] [config] transformer-guided-alignment-layer: last
[2020-06-23 03:56:35] [config] transformer-heads: 8
[2020-06-23 03:56:35] [config] transformer-no-projection: false
[2020-06-23 03:56:35] [config] transformer-postprocess: dan
[2020-06-23 03:56:35] [config] transformer-postprocess-emb: d
[2020-06-23 03:56:35] [config] transformer-preprocess: ""
[2020-06-23 03:56:35] [config] transformer-tied-layers:
[2020-06-23 03:56:35] [config]   []
[2020-06-23 03:56:35] [config] transformer-train-position-embeddings: false
[2020-06-23 03:56:35] [config] type: transformer
[2020-06-23 03:56:35] [config] ulr: false
[2020-06-23 03:56:35] [config] ulr-dim-emb: 0
[2020-06-23 03:56:35] [config] ulr-trainable-transformation: false
[2020-06-23 03:56:35] [config] version: v1.8.2 2111c28 2019-10-16 08:36:48 -0700
[2020-06-23 03:56:35] [config] vocabs:
[2020-06-23 03:56:35] [config]   - /usr/src/app/models/en-mr/opus.spm32k-spm32k.vocab.yml
[2020-06-23 03:56:35] [config]   - /usr/src/app/models/en-mr/opus.spm32k-spm32k.vocab.yml
[2020-06-23 03:56:35] [config] weights:
[2020-06-23 03:56:35] [config]   []
[2020-06-23 03:56:35] [config] word-penalty: 0
[2020-06-23 03:56:35] [config] word-scores: false
[2020-06-23 03:56:35] [config] workspace: 512
[2020-06-23 03:56:35] [config] Loaded model has been created with Marian v1.8.2 2111c28 2019-10-16 08:36:48 -0700
[2020-06-23 03:56:35] [data] Loading vocabulary from JSON/Yaml file /usr/src/app/models/en-mr/opus.spm32k-spm32k.vocab.yml
[2020-06-23 03:56:35] [data] Loading vocabulary from JSON/Yaml file /usr/src/app/models/en-es/opus.bpe32k-bpe32k.vocab.yml
[2020-06-23 03:56:36] [data] Loading vocabulary from JSON/Yaml file /usr/src/app/models/en-fi/opus.spm32k-spm32k.vocab.yml
[2020-06-23 03:56:36] [data] Loading vocabulary from JSON/Yaml file /usr/src/app/models/en-ml/opus+bt.spm32k-spm32k.vocab.yml
[2020-06-23 03:56:36] [memory] Extending reserved space to 512 MB (device cpu0)
[2020-06-23 03:56:36] Loading scorer of type transformer as feature F0
[2020-06-23 03:56:36] Loading model from /usr/src/app/models/en-es/opus.bpe32k-bpe32k.transformer.model1.npz.best-perplexity.npz
[2020-06-23 03:56:37] [data] Loading vocabulary from JSON/Yaml file /usr/src/app/models/en-mr/opus.spm32k-spm32k.vocab.yml
[2020-06-23 03:56:37] [memory] Extending reserved space to 512 MB (device cpu0)
[2020-06-23 03:56:37] Loading scorer of type transformer as feature F0
[2020-06-23 03:56:37] Loading model from /usr/src/app/models/en-fi/opus.spm32k-spm32k.transformer-align.model1.npz.best-perplexity.npz
[2020-06-23 03:56:38] [memory] Extending reserved space to 512 MB (device cpu0)
[2020-06-23 03:56:38] Loading scorer of type transformer as feature F0
[2020-06-23 03:56:38] Loading model from /usr/src/app/models/en-ml/opus+bt.spm32k-spm32k.transformer-align.model1.npz.best-perplexity.npz
[2020-06-23 03:56:38] [memory] Extending reserved space to 512 MB (device cpu0)
[2020-06-23 03:56:38] Loading scorer of type transformer as feature F0
[2020-06-23 03:56:38] Loading model from /usr/src/app/models/en-mr/opus.spm32k-spm32k.transformer-align.model1.npz.best-perplexity.npz
[2020-06-23 03:56:39] Server is listening on port 10001
[2020-06-23 03:56:39] Server is listening on port 10002
[2020-06-23 03:56:40] Server is listening on port 10003
[2020-06-23 03:56:40] Server is listening on port 10004

santhoshtr avatar Jun 23 '20 04:06 santhoshtr

Sorry, my comment was misleading. The docker run works fine. docker-compose up does not (my log is from compose)

y-bo avatar Jun 23 '20 12:06 y-bo

Sorry, my comment was misleading. The docker run works fine. docker-compose up does not (my log is from compose)

Perhaps you need to check what is the problem with docker-compose script? Anyway, it is independent of the issue being discussed here, right? Let me know if it is not.

santhoshtr avatar Jun 24 '20 04:06 santhoshtr

Any updates on progress perhaps?

xirect avatar Sep 02 '20 20:09 xirect

Any updates on progress perhaps?

https://github.com/Helsinki-NLP/Opus-MT/pull/22 has the fix for this and I posted the log of that working fine. In case that is not helpful, please let us know what exact issue you see.

santhoshtr avatar Sep 04 '20 09:09 santhoshtr

I had a lot of other issues which are now fixed.

xirect avatar Sep 04 '20 11:09 xirect

I had a lot of other issues which are now fixed.

https://hub.docker.com/r/veer66/opus-mt-th-en This third-party docker image works fine for me，have a try？

jotline avatar Nov 03 '20 02:11 jotline