the accuracy is very low when using my own data training models(on board)

[lancezhangsf]

Hi, when using my own data for training model, and instand of the ds_cnn_weights.h , recompile the kws_realtime_test.bin ,but on board(stm32f746) ,it's difficult to recongize. but i test the model on pc, the accuracy is very high.

python test.py --model_architecture ds_cnn --model_size_info 5 64 10 4 2 2 64 3 3 1 1 64 3 3 1 1 64 3 3 1 1 64 3 3 1 1 --dct_coefficient_count 10 --window_size_ms 40 --window_stride_ms 20 --checkpoint /opt/zsf_workspace/WorkSpace/SoundWorkSpace/ML-KWS-for-MCU/work/DS_CNN/DS_CNN1/training/best/ds_cnn_9750.ckpt-30000 2019-03-07 14:18:00.541952: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2019-03-07 14:18:00.571174: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2499835000 Hz 2019-03-07 14:18:00.574544: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x41ee720 executing computations on platform Host. Devices: 2019-03-07 14:18:00.574588: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): , WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/confusion_matrix.py:193: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/confusion_matrix.py:194: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to check for files with this prefix. INFO:tensorflow:Restoring parameters from /opt/zsf_workspace/WorkSpace/SoundWorkSpace/ML-KWS-for-MCU/work/DS_CNN/DS_CNN1/training/best/ds_cnn_9750.ckpt-30000 INFO:tensorflow:set_size=5686 INFO:tensorflow:Confusion Matrix: [[505 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 512 0 0 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 259 0 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 260 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 263 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 233 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 512 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 0 343 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 0 0 529 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 0 0 0 253 0 0 0 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 454 0 0 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 283 0 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 383 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 521 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 376]] INFO:tensorflow:Training accuracy = 100.00% (N=5686) INFO:tensorflow:set_size=721 INFO:tensorflow:Confusion Matrix: [[66 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 61 0 0 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 31 1 0 0 0 0 0 1 0 0 0 0 0] [ 0 0 0 1 35 0 1 0 0 0 0 0 0 0 0 0] [ 0 0 2 0 2 28 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 1 0 35 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 74 0 0 0 0 0 0 0 1] [ 0 0 0 0 0 0 0 1 31 0 0 0 0 0 0 1] [ 0 0 1 0 0 0 0 0 0 53 0 0 0 0 0 0] [ 0 0 2 0 0 0 0 0 0 0 26 0 2 0 0 0] [ 0 0 0 0 0 2 0 0 0 0 0 70 0 0 0 0] [ 0 0 0 0 0 1 0 0 0 0 0 0 25 0 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 47 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 67 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 52]] INFO:tensorflow:Validation accuracy = 97.23% (N=721) INFO:tensorflow:set_size=729 INFO:tensorflow:Confusion Matrix: [[67 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 59 0 0 0 1 0 0 0 0 0 0 1 0 0] [ 0 0 0 31 1 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 19 0 1 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 2 29 0 0 0 0 0 0 1 0 0 0] [ 0 0 0 0 1 0 39 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 51 0 0 0 0 0 0 1 0] [ 0 0 0 0 0 0 0 1 56 0 0 0 0 0 0 0] [ 0 0 1 0 0 0 0 0 0 59 0 0 1 0 0 0] [ 0 0 0 0 0 0 0 0 0 2 27 0 0 0 0 0] [ 0 0 0 0 0 1 0 0 0 0 0 70 0 0 0 0] [ 0 0 0 0 0 0 0 0 0 3 0 0 39 0 0 0] [ 0 0 1 0 0 0 0 0 0 1 0 2 0 56 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 54 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 51]] INFO:tensorflow:Test accuracy = 96.98% (N=729)

root@ubun:/opt/zsf_workspace/WorkSpace/SoundWorkSpace/ML-KWS-for-MCU# python quant_test.py --model_architecture ds_cnn --model_size_info 5 64 10 4 2 2 64 3 3 1 1 64 3 3 1 1 64 3 3 1 1 64 3 3 1 1 --dct_coefficient_count 10 --window_size_ms 40 --window_stride_ms 20 --checkpoint /opt/zsf_workspace/WorkSpace/SoundWorkSpace/ML-KWS-for-MCU/work/DS_CNN/DS_CNN1/training/best/ds_cnn_9750.ckpt-30000_bnfused --act_max 32 0 0 0 0 0 0 0 0 0 0 0 2019-03-07 14:47:18.762088: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2019-03-07 14:47:18.787024: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2499835000 Hz 2019-03-07 14:47:18.790068: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x451b6b0 executing computations on platform Host. Devices: 2019-03-07 14:47:18.790111: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): , WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/confusion_matrix.py:193: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/confusion_matrix.py:194: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to check for files with this prefix. INFO:tensorflow:Restoring parameters from /opt/zsf_workspace/WorkSpace/SoundWorkSpace/ML-KWS-for-MCU/work/DS_CNN/DS_CNN1/training/best/ds_cnn_9750.ckpt-30000_bnfused DS-CNN_conv_1_weights_0 number of wts/bias: (10, 4, 1, 64) dec bits: 10 max: (0.078125,0.07804396) min: (-0.05859375,-0.05906091) DS-CNN_conv_1_biases_0 number of wts/bias: (64,) dec bits: 8 max: (0.3125,0.31323364) min: (-0.23828125,-0.23821199) DS-CNN_conv_ds_1_dw_conv_depthwise_weights_0 number of wts/bias: (3, 3, 64, 1) dec bits: 5 max: (3.5625,3.5662289) min: (-3.15625,-3.1511166) DS-CNN_conv_ds_1_dw_conv_biases_0 number of wts/bias: (64,) dec bits: 6 max: (1.390625,1.3905205) min: (-1.0625,-1.0663435) DS-CNN_conv_ds_1_pw_conv_weights_0 number of wts/bias: (1, 1, 64, 64) dec bits: 7 max: (0.84375,0.84409374) min: (-0.8125,-0.81246847) DS-CNN_conv_ds_1_pw_conv_biases_0 number of wts/bias: (64,) dec bits: 6 max: (1.015625,1.0175782) min: (-1.0625,-1.0584457) DS-CNN_conv_ds_2_dw_conv_depthwise_weights_0 number of wts/bias: (3, 3, 64, 1) dec bits: 5 max: (2.40625,2.4150395) min: (-2.34375,-2.3446453) DS-CNN_conv_ds_2_dw_conv_biases_0 number of wts/bias: (64,) dec bits: 6 max: (1.40625,1.4129168) min: (-0.96875,-0.96886146) DS-CNN_conv_ds_2_pw_conv_weights_0 number of wts/bias: (1, 1, 64, 64) dec bits: 7 max: (0.71875,0.7201497) min: (-0.875,-0.8777769) DS-CNN_conv_ds_2_pw_conv_biases_0 number of wts/bias: (64,) dec bits: 6 max: (1.765625,1.7612783) min: (-1.1875,-1.1944273) DS-CNN_conv_ds_3_dw_conv_depthwise_weights_0 number of wts/bias: (3, 3, 64, 1) dec bits: 6 max: (1.625,1.6269287) min: (-1.5,-1.4997675) DS-CNN_conv_ds_3_dw_conv_biases_0 number of wts/bias: (64,) dec bits: 6 max: (1.5,1.5054036) min: (-1.375,-1.3712918) DS-CNN_conv_ds_3_pw_conv_weights_0 number of wts/bias: (1, 1, 64, 64) dec bits: 7 max: (0.9140625,0.9122335) min: (-0.875,-0.87133324) DS-CNN_conv_ds_3_pw_conv_biases_0 number of wts/bias: (64,) dec bits: 5 max: (1.25,1.2655265) min: (-2.15625,-2.146341) DS-CNN_conv_ds_4_dw_conv_depthwise_weights_0 number of wts/bias: (3, 3, 64, 1) dec bits: 6 max: (1.90625,1.9060508) min: (-1.671875,-1.6707895) DS-CNN_conv_ds_4_dw_conv_biases_0 number of wts/bias: (64,) dec bits: 6 max: (1.015625,1.0225885) min: (-1.28125,-1.2750894) DS-CNN_conv_ds_4_pw_conv_weights_0 number of wts/bias: (1, 1, 64, 64) dec bits: 6 max: (0.984375,0.97935057) min: (-1.078125,-1.0792036) DS-CNN_conv_ds_4_pw_conv_biases_0 number of wts/bias: (64,) dec bits: 6 max: (1.6875,1.6924481) min: (0.0,0.0039412514) DS-CNN_fc1_weights_0 number of wts/bias: (64, 16) dec bits: 6 max: (1.25,1.2544212) min: (-1.53125,-1.5301858) DS-CNN_fc1_biases_0 number of wts/bias: (16,) dec bits: 8 max: (0.25390625,0.25582832) min: (-0.21875,-0.2186358) INFO:tensorflow:set_size=5686 INFO:tensorflow:Confusion Matrix: [[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 536] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 499 0 0 0 0 0 0 0 0 1 0 0 0 0] [ 0 0 0 261 0 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 255 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 271 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 273 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 491 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 0 335 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 0 0 460 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 0 0 0 275 0 0 0 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 508 0 0 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 243 0 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 396 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 520 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 362]] INFO:tensorflow:Training accuracy = 90.56% (N=5686) INFO:tensorflow:set_size=721 INFO:tensorflow:Confusion Matrix: [[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 66] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 60 0 0 0 0 0 0 0 0 0 0 1 0 0] [ 0 0 0 31 1 0 0 0 0 0 0 0 1 0 0 0] [ 0 0 0 1 35 0 1 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 32 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 1 0 35 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 73 0 0 0 0 0 0 1 1] [ 0 0 0 0 0 0 0 1 31 0 0 0 0 0 0 1] [ 0 0 1 0 0 0 0 0 0 53 0 0 0 0 0 0] [ 0 0 2 0 0 1 0 0 0 1 24 0 2 0 0 0] [ 0 0 0 0 0 3 0 0 0 0 0 69 0 0 0 0] [ 0 0 0 0 0 1 0 0 0 0 0 0 25 0 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 47 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 67 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 52]] INFO:tensorflow:Validation accuracy = 87.93% (N=721) INFO:tensorflow:set_size=729 INFO:tensorflow:Confusion Matrix: [[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 67] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 57 0 0 0 1 0 0 0 0 0 0 3 0 0] [ 0 0 0 31 1 0 0 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 19 0 1 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 2 29 0 0 0 0 0 0 1 0 0 0] [ 0 0 0 0 1 0 39 0 0 0 0 0 0 0 0 0] [ 0 0 0 0 0 0 0 51 0 0 0 0 0 0 0 1] [ 0 0 0 0 0 0 0 1 56 0 0 0 0 0 0 0] [ 0 0 1 0 0 0 0 0 0 59 0 0 1 0 0 0] [ 0 0 0 0 0 0 0 0 0 2 27 0 0 0 0 0] [ 0 0 0 1 0 1 0 0 0 0 0 69 0 0 0 0] [ 0 0 0 0 0 0 0 0 0 4 0 0 38 0 0 0] [ 0 0 0 0 0 0 0 0 0 1 0 2 0 57 0 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 54 0] [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 51]] INFO:tensorflow:Test accuracy = 87.38% (N=729)

[bkanakietac]

in the quant_test.py, all the values of --act_max 32 0 0 0 0 0 0 0 0 0 0 0 has to be some power of two instead of zeros. Check out the quant_guide and re-read the quantization section again.

You need to calculate the bias shift and out shift for each layer and then put them in the cpp source code. It is not trivial, but definitely works.

Also check #68 , #51 and try reproducing the simple DNN example first. We all want high accuracy results but if you figure out one simple, it would be easier to extend the idea to any model in this work.

[bkanaki]

first of all, good that you figured some stuff. secondly, for the right shift, there is a problem. I had the same issue and logged #90 but no one responded.

So input to your last layer is Q5.2 and weights Q1.6 but bias is Qx.10. Try changing this input layer to Q3.4 and see if the accuracy is still maintained (or maybe better?) and then your bias shift will be zero. I did something similar.

[ccnankai]

@bkanaki Thank you for your kind help.

[bkanakietac]

Read, experiment, repeat! Did you read the quantization guide above mentioned? It is already decently documented with an example. All the relevant code is also in this repository. Explore! The answer to your last question is in your previous comment.

As I had mentioned previously, it would be a good start to experiment with the DNN model because of its simplicity, but everyone just wants to use dscnn. 😒

[wayne175]

@bkanaki Thank you for your kind help.

I've had the same problem. how can I solve this problem?

[wayne175]

first of all, good that you figured some stuff. secondly, for the right shift, there is a problem. I had the same issue and logged #90 but no one responded.

So input to your last layer is Q5.2 and weights Q1.6 but bias is Qx.10. Try changing this input layer to Q3.4 and see if the accuracy is still maintained (or maybe better?) and then your bias shift will be zero. I did something similar.

I've had the same problem. how can I solve this problem?

[saichand07]

Read, experiment, repeat! Did you read the quantization guide above mentioned? It is already decently documented with an example. All the relevant code is also in this repository. Explore! The answer to your last question is in your previous comment.

As I had mentioned previously, it would be a good start to experiment with the DNN model because of its simplicity, but everyone just wants to use dscnn. 😒

@bkanakietac thank you for this comment,

I have calculated the bias shift and out_shift of each layer and I did put them in cpp,the only DNN model is working well but the DS_CNN model is not detecting the words properly at all, I also tuned the recording window, averaging window len, detection threshold if you have any idea, could you please tell me what else still i need to consider.

[JeffyCN]

first of all, good that you figured some stuff. secondly, for the right shift, there is a problem. I had the same issue and logged #90 but no one responded. So input to your last layer is Q5.2 and weights Q1.6 but bias is Qx.10. Try changing this input layer to Q3.4 and see if the accuracy is still maintained (or maybe better?) and then your bias shift will be zero. I did something similar.

I've had the same problem. how can I solve this problem?

Seems like it's not supported: https://github.com/ARM-software/CMSIS_5/issues/327#issuecomment-369690102

Adjust act_max might avoid that.