Jonathan Shen

As the error says it seems the EmbLookupDefaultTheta cannot be made with int32 dtype. Try casting emb_ids to float32?

Hmm... To check if EmbLookupDefaultTheta is actually the problem, can you replace that line with `curr_emb = tf.zeros(expected_size)` and see if everything runs fine?

Yes but from the message it seems to be a problem with embedding_lookup but tf.nn.embedding_lookup should be supported on GPU.

Yes, that is exactly the problem, except that I thought tf.nn.embedding_lookup was supposed to work on GPU. Otherwise, if it is not possible to use tf.nn.embedding_lookup inside Recurrent on GPU,...

Hm, this is strange. First, can you check that it works on cpu: ``` $ bazel clean $ bazel build -c opt lingvo:trainer (note: no --config=cuda) $ CUDA_VISIBLE_DEVICES= bazel-bin/lingvo/trainer --run_locally=cpu...

The CPU segfault is weird. can you see if this test passes bazel test -c opt //lingvo/core/ops:beam_search_step_op_test

Ok, I think this might be this compiler mismatch problem with the latest tf-nightly build that we ran into last week. Can you try pulling the latest version of lingvo,...

The packages inside your server is not important, the packages inside docker is what matters. The docker file has been updated to g++7 here: https://github.com/tensorflow/lingvo/blob/29099ef71c9d9eac66e35ff27371479f284c0c7a/docker/dev.dockerfile#L27 So hopefully if you just...

Yes, docker build again. Make sure to set --no-cache so it doesn't try to use cached packages.

Can you try the bazel test -c opt //lingvo/core/ops:beam_search_step_op_test again? If it still fails something is going wrong. In that case please run https://github.com/tensorflow/lingvo/blob/master/tf_env_collect.sh and paste the outputs. If that...