Error: DefaultVocabulary file is expected to contain an entry for (...)

[sdlmw]

`[2019-11-25 21:28:58] [config] tied-embeddings-all: false [2019-11-25 21:28:58] [config] tied-embeddings-src: false [2019-11-25 21:28:58] [config] train-sets: [2019-11-25 21:28:58] [config] - /191030/1_ForTrain/train.tok.en [2019-11-25 21:28:58] [config] - /191030/1_ForTrain/train.tok.zh [2019-11-25 21:28:58] [config] transformer-aan-activation: swish [2019-11-25 21:28:58] [config] transformer-aan-depth: 2 [2019-11-25 21:28:58] [config] transformer-aan-nogate: false [2019-11-25 21:28:58] [config] transformer-decoder-autoreg: self-attention [2019-11-25 21:28:58] [config] transformer-dim-aan: 2048 [2019-11-25 21:28:58] [config] transformer-dim-ffn: 2048 [2019-11-25 21:28:58] [config] transformer-dropout: 0 [2019-11-25 21:28:58] [config] transformer-dropout-attention: 0 [2019-11-25 21:28:58] [config] transformer-dropout-ffn: 0 [2019-11-25 21:28:58] [config] transformer-ffn-activation: swish [2019-11-25 21:28:58] [config] transformer-ffn-depth: 2 [2019-11-25 21:28:58] [config] transformer-guided-alignment-layer: last [2019-11-25 21:28:58] [config] transformer-heads: 8 [2019-11-25 21:28:58] [config] transformer-no-projection: false [2019-11-25 21:28:58] [config] transformer-postprocess: dan [2019-11-25 21:28:58] [config] transformer-postprocess-emb: d [2019-11-25 21:28:58] [config] transformer-preprocess: "" [2019-11-25 21:28:58] [config] transformer-tied-layers: [2019-11-25 21:28:58] [config] [] [2019-11-25 21:28:58] [config] type: amun [2019-11-25 21:28:58] [config] ulr: false [2019-11-25 21:28:58] [config] ulr-dim-emb: 0 [2019-11-25 21:28:58] [config] ulr-dropout: 0 [2019-11-25 21:28:58] [config] ulr-keys-vectors: "" [2019-11-25 21:28:58] [config] ulr-query-vectors: "" [2019-11-25 21:28:58] [config] ulr-softmax-temperature: 1 [2019-11-25 21:28:58] [config] ulr-trainable-transformation: false [2019-11-25 21:28:58] [config] valid-freq: 10000u [2019-11-25 21:28:58] [config] valid-max-length: 1000 [2019-11-25 21:28:58] [config] valid-metrics: [2019-11-25 21:28:58] [config] - cross-entropy [2019-11-25 21:28:58] [config] valid-mini-batch: 32 [2019-11-25 21:28:58] [config] vocabs: [2019-11-25 21:28:58] [config] - /191030/2_ForTune/Tune.en [2019-11-25 21:28:58] [config] - /191030/2_ForTune/Tune.zh [2019-11-25 21:28:58] [config] word-penalty: 0 [2019-11-25 21:28:58] [config] workspace: 2048 [2019-11-25 21:28:58] [config] Model is being created with Marian v1.7.6 1d4ba73 2019-05-11 17:16:31 +0100 [2019-11-25 21:28:58] Using single-device training [2019-11-25 21:28:58] [data] Loading vocabulary from text file /191030/2_ForTune/Tune.en [2019-11-25 21:28:58] Error: DefaultVocabulary file /191030/2_ForTune/Tune.en is expected to contain an entry for [2019-11-25 21:28:58] Error: Aborted from marian::DefaultVocab::load(const string&, size_t)::<lambda(const string&, const string&, marian::Word)> in /marian/src/data/default_vocab.cpp:154

[CALL STACK] [0x592ded] [0x594c0c] [0x586ed5] [0x587c7b] [0x59cccc] [0x5a8771] [0x4d071d] [0x4f9392] [0x42e213] [0x40c0da] [0x7f8724bb9830] __libc_start_main + 0xf0 [0x42b7f9] `

I got this mistake while training the engine. Please I ask how to solve this problem

[emjotde]

Hi, did you create the vocabulary somehow by hand? It seems to be missing some symbols.

[sdlmw]

Hi emjotde, I just simply extracted part from the train set as the vocabulary，

[emjotde]

You would need to use the marian_vocab binary for this. Marian requires a number of special symbols, if those are missing you will get errors like this.

[sdlmw]

ok, I will retry usage: ./marian/build/marian-vocab [OPTIONS]

[snukky]

Closing, the question has been answered.

[frankseide]

Actually no need to use marian_vocab. You just need these three entries, and by convention at the start of the vocab:

<unk>
<s>
</s>

I create my vocabs with something like this:

echo -e '<unk>\n<s>\n</s>' > VOCAB
cat CORPUS \
  | tr ' \r' '\n' \
  | sort -u | grep . \
  >> VOCAB

I believe marian_vocab creates a JSON file. JSON is not a suitable format for representing vocabularies, because JSON implies UTF-8 encoding. That is problematic because

• I found corpora do contain invalid UTF-8, causing mal-formed JSON files;
• Marian is encoding-agnostic.

[sappho192]

Actually no need to use marian_vocab. You just need these three entries, and by convention at the start of the vocab:

<unk>
<s>
</s>

@frankseide Thank you so much! You saved my day!