Error running with TF 0.12.1, Python 3.4.3, Ubuntu 14.04

[dab3-2014]

Hi...So I installed all the required dependencies w/ pip3, made sure I can import them in Python3 with no issues, and downloaded dataset as described. Now running: $ python3 udc_train.py produces the following error:

InvalidArgumentError (see above for traceback): Incompatible shapes: [80,1] vs. [160,1]
         [[Node: prediction/logistic_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](prediction/Squeeze, prediction/ToFloat)]]
         [[Node: recall_at_2/ToInt64/_91 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_217_recall_at_2/ToInt64", tensor_type=DT_INT64, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Any idea why this is happening? Any idea how to fix it?

Thanks.

[hjguyhan]

I have same error running with TF 0.12.1, python 2.7, ubuntu 16.04, gtx 1070, cuda 8.0, cudnn 5.1.5. So I ran it on AWS (g2.2xlarge) and the same problem occurred. It is like an error that occurs in tensorflow. But this issue was already closed. https://github.com/dennybritz/chatbot-retrieval/issues/15

[ghost]

Commenting out monitors seems to be fixing the issue. Please confirm?

[vaibhavravi]

@pavelromashkin Yes this helped me. Probably because I'm using TF 1.0 where the monitors are deprecated.

Any advice on how to evaluate by reimplementing the streaming in TF 1.0 without monitors?

[pengcao]

我使用的是tensorflow-1.12把udc_hparams.py下面的代码 tf.flags.DEFINE_integer("batch_size", 128, "Batch size during training") tf.flags.DEFINE_integer("eval_batch_size", 16, "Batch size during evaluation") 改成如下形式就行了 tf.flags.DEFINE_integer("batch_size", 64, "Batch size during training") tf.flags.DEFINE_integer("eval_batch_size", 8, "Batch size during evaluation")

[pengcao]

I am using tensorflow=1.12.0 on win7 64. i am running into this error during training. thanks. InvalidArgumentError (see above for traceback): Incompatible shapes: [20,1] vs. [80,1] [[node prediction/logistic_loss/mul (defined at F:\09.Practice\chatbot-deeplearning-retrieval\models\dual_encoder.py:87) = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](prediction/Squeeze, prediction/ToFloat)]]