High overhead of power measurements for CPU inference measurements

[psyhtest]

During the MLPerf Inference v1.0 round, I noticed that the power workflow when used with CPU inference occasionally seemed to incur a rather high overhead (~10%), for example:

• Xavier with power measurements:

  • ArmNN v21.02: 68.5 ms
  • TFLite v2.4.1: 78.0 ms

• Xavier without power measurements (compliance):

  • ArmNN v21.02: 60.2 ms
  • TFLite v2.4.1: 78.5 ms

Here, ArmNN is faster than TFLite but takes a big hit under the power workflow. TFLite, however, is not affected.