SIN icon indicating copy to clipboard operation
SIN copied to clipboard

How long does it take to train? Has anyone trained successfully?

Open ghost opened this issue 6 years ago • 5 comments

ghost avatar Sep 29 '18 00:09 ghost

I found the calculation of edges costs most time in training.

ghost avatar Sep 29 '18 08:09 ghost

I tried to run the training on my own dataset, but got an issue: ZeroDivisionError: float division by zero

UnknownError (see above for traceback): exceptions.ZeroDivisionError: float division by zero [[Node: edges/PyFunc = PyFunc[Tin=[DT_FLOAT, DT_FLOAT], Tout=[DT_FLOAT], token="pyfunc_4", _device="/job:localhost/replica:0/task:0/cpu:0"](roi-data/rois/_141, _arg_Placeholder_1_0_1)]] [[Node: edges/PyFunc/_143 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1461_edges/PyFunc", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

Still confusing... Could u please help?

BestSongEver avatar Sep 29 '18 13:09 BestSongEver

I tried to run the training on my own dataset, but got an issue: ZeroDivisionError: float division by zero

UnknownError (see above for traceback): exceptions.ZeroDivisionError: float division by zero [[Node: edges/PyFunc = PyFunc[Tin=[DT_FLOAT, DT_FLOAT], Tout=[DT_FLOAT], token="pyfunc_4", _device="/job:localhost/replica:0/task:0/cpu:0"](roi-data/rois/_141, _arg_Placeholder_1_0_1)]] [[Node: edges/PyFunc/_143 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1461_edges/PyFunc", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

Still confusing... Could u please help? emmm, I re-implemented it with Pytorch and training it on VOC2007. There is not error when training except the speed is so slow. Maybe you could locate your error and add a small constant.

ghost avatar Sep 30 '18 02:09 ghost

No, I met the problem with incompatible shape.So sad.

Elasine avatar Nov 27 '18 13:11 Elasine

No, I met the problem with incompatible shape.So sad.

Hi Elasine, I have the same problem as you. Did you have a clue to solve it?

zhl95 avatar Dec 23 '18 05:12 zhl95