serving deepfm error under high performance (fc_layer matrix null)

[mattxia]

1.using 400 concurrent 2.by gRPC 3.default deepfm model and also some production model(61 field, 9xxx dim) 4.some error occured 5.and we stepped in and find some times fc_layer is null

[wangcaihua]

What do you mean by "sometimes"?

[wangcaihua]

can you post you graph json file?

[mattxia]

What do you mean by "sometimes"?

because some request reponse with error, so I think the null occurs sometimes

[mattxia]

can you post you graph json file?

we used the local deepfm model, and graph.json { "data" : { "format" : "libsvm", "indexrange" : 148, "numfield" : 13, "validateratio" : 0.1, "sampleratio" : 0.2, "useshuffle" : false, "posnegratio" : -1.0, "translabel" : "NoTrans" }, "model" : { "modeltype" : "T_FLOAT_DENSE", "modelsize" : 148, "blockSize" : 1000000 }, "train" : { "epoch" : 5, "numupdateperepoch" : 10, "batchsize" : 128, "lr" : 0.5, "decayclass" : "WarmRestarts", "decayalpha" : 0.05, "decaybeta" : 0.001 }, "layers" : [ { "name" : "input", "type" : "SimpleInputLayer", "outputdim" : 1, "transfunc" : { "type" : "Identity" }, "optimizer" : { "type" : "Momentum", "momentum" : 0.9 } }, { "name" : "embedding", "type" : "Embedding", "outputdim" : 104, "numfactors" : 8, "optimizer" : { "type" : "Momentum", "momentum" : 0.9 } }, { "name" : "innerSumPooling", "type" : "BiInnerSumCross", "outputdim" : 1, "inputlayer" : "embedding" }, { "name" : "fclayer", "type" : "FCLayer", "outputdims" : [ 100, 100, 1 ], "inputlayer" : "embedding", "transfuncs" : [ { "type" : "Relu" }, { "type" : "Relu" }, { "type" : "Identity" } ], "optimizer" : { "type" : "Momentum", "momentum" : 0.9 } }, { "name" : "sumPooling", "type" : "SumPooling", "outputdim" : 1, "inputlayers" : [ "input", "innerSumPooling", "fclayer" ] }, { "name" : "simpleLossLayer", "type" : "SimpleLossLayer", "outputdim" : -1, "inputlayer" : "sumPooling", "lossfunc" : { "type" : "LogLoss" } } ] }