DeepCTR icon indicating copy to clipboard operation
DeepCTR copied to clipboard

训练时正则loss变成nan了是怎么回事?

Open chenjinting123 opened this issue 4 years ago • 3 comments

发现loss为nan后,我把每个batch总的loss,bce loss, acc都打印出来,发现总的loss为Nan, bce loss不为nan。 正则项定义用的是默认的参数,模型定义:model = xDeepFM(linear_feature_columns,dnn_feature_columns, dnn_use_bn=True, task='binary')

batch: 383 0.6420018 0.6280691 0.7282944 batch: 384 0.6263658 0.6280647 0.7282823 batch: 385 0.6214759 0.6280476 0.72826767 batch: 386 0.6920916 0.62821305 0.72858125 batch: 387 nan 0.6595617 0.72703314 batch: 388 nan 0.69080186 0.72551566 batch: 389 nan 0.7215608 0.7240385 batch: 390 nan 0.7521064 0.7225713 batch: 391 nan 0.78246844 0.7210768 batch: 392 nan 0.8138485 0.7195303 batch: 393 nan 0.8465953 0.71797174 batch: 394 nan 0.8777735 0.71651256 batch: 395 nan 0.90917253 0.7150361 batch: 396 nan 0.9399229 0.7135991 batch: 397 nan 0.97097033 0.7121398 batch: 398 nan 1.0026878 0.710634 batch: 399 nan 1.0329365 0.7092212 batch: 400 nan 1.063781 0.7077667 batch: 401 nan 1.0945466 0.7063146 batch: 402 nan 1.1250108 0.7048794 batch: 403 nan 1.1550279 0.70347065 batch: 404 nan 1.1851554 0.702052 batch: 405 nan 1.214987 0.7006499

chenjinting123 avatar Aug 05 '20 06:08 chenjinting123

loss为nan,bce loss不为nan,调小正则系数看看

shenweichen avatar Aug 09 '20 12:08 shenweichen

loss为nan,将pred结果转换为np.float64即可。如下: pred_ans = model.predict(test_model_input, batch_size=256) pred_ans = pred_ans.astype(np.float64)

yuanzhiyu avatar Aug 12 '20 04:08 yuanzhiyu

你好,请问怎么打印的loss,我这啥都不打印

1980695671 avatar May 08 '21 09:05 1980695671