inference times too large

[Anubhav7]

hi

I am getting an inference time of around .8s on GTX 1070 (8GB). The image sizes are 1080x1920. Cuda version is 9.0. What may I be doing wrong?

[hellochick]

Hey, it will initialize the model at first iteration, so the time you calculated contain the time of initialization. You can try to run more images and calculate like following:

for i in range(10):
     start_time = time.time()
     preds = sess.run(pred, feed_dict={x: img})
     end_time = time.time()
     
    print('time: ', end_time - start_time)

[Anubhav7]

Hey,

Thanks for the quick reply. I tried your suggestion and now it reports .06s per frame. My analysis says that the function decode_labels is taking the rest of the time. I am thinking of ways to speed that up.. do you have any suggestions?

[hellochick]

Hey @Anubhav7 , I will fix this bottleneck tomorrow, just wait for me, thanks.

[Fengmoon93]

I just resize the cityscapes dataset into 512*1024,I record the time that the network processed the data,the average time is 0.02s,but when it loads the checkpoint flie,the first iteration takes about 0.04s. Anyway,thanks for your amazing job~ @hellochick

[hellochick]

@Anubhav7, I have change the decode_label function. Can you test on it again?

[Anubhav7]

You beat me to the fix :). Thanks for the help, really appreciate it. I am getting inference times of around .07s now

[StOnEGiggity]

Hi, thanks for your excellent work at first.There is one thing confuses me that inference time doesn't include interp layer only when testing.In the official code or paper, it looks like inference time includes this part and it costs only 0.5ms in caffe.

[hellochick]

Hey @StOnEGiggity. My way to calculate inference time is to subtract the time of processing input data, so it contains interp layer. Does this confuse you, or something else?

Btw, it seems that caffe use caffe time to calculate, and it only contains the time of forwarding in the network.

[StOnEGiggity]

Sorry for my unsuitable expression. I mean that the time may contain this line https://github.com/hellochick/ICNet-tensorflow/blob/4aeb4005a2c25636d4d9bb4914f1d641f591d7cf/evaluate.py#L100

[hellochick]

@StOnEGiggity , Ohhh!!!!! I got it, this is my fault. I forgot to calculate this time at evaluate.py, now i understand what you mean. Thanks for your reply, I have just fixed it .

[StOnEGiggity]

Hi, I am interested in your new inference time.I have larger one than paper refers to due to GPUs.Would you mind updating it somewhere?Thanks a lot:) @hellochick

[hellochick]

Hey @StOnEGiggity , the new inference time test on GTX 1080ti shown below: Inference time excluding loading data ( Calculated as described in paper ): ~0.03s Inference time including loading data : ~0.04s