bigwheels icon indicating copy to clipboard operation
bigwheels copied to clipboard

LiveMetric: realtime statistics for debug UI.

Open hysw opened this issue 1 year ago • 9 comments

This PR is a fork of #377 .

Adding real time statistics into the metric framework so it is easier to read.

hysw avatar Jan 16 '24 00:01 hysw

Could you please add a quick summary of what problem the PR solves? I am not extremely familiar with the benchmark, and I have not touched BigWheels for a few months.

mdagois avatar Jan 16 '24 04:01 mdagois

Could you please add a quick summary of what problem the PR solves? I am not extremely familiar with the benchmark, and I have not touched BigWheels for a few months.

This gives a readable statistics for cpu submission time / gpu work duration / etc. i.e. the leading digit does not jump around every frame. This version is due to suggestion of @Keenuts 's suggestion on have it in the metric framework. The stand alone version is #377 .

hysw avatar Jan 16 '24 05:01 hysw

Could you please add a quick summary of what problem the PR solves? I am not extremely familiar with the benchmark, and I have not touched BigWheels for a few months.

This gives a readable statistics for cpu submission time / gpu work duration / etc. i.e. the leading digit does not jump around every frame. This version is due to suggestion of @Keenuts 's suggestion on have it in the metric framework. The stand alone version is #377 .

It is a kind of tracker of statistics for an underlying gauge metrics, and you reset it (by clearing history) whenever you want. Is that correct?

mdagois avatar Jan 17 '24 08:01 mdagois

Could you please add a quick summary of what problem the PR solves? I am not extremely familiar with the benchmark, and I have not touched BigWheels for a few months.

This gives a readable statistics for cpu submission time / gpu work duration / etc. i.e. the leading digit does not jump around every frame. This version is due to suggestion of @Keenuts 's suggestion on have it in the metric framework. The stand alone version is #377 .

It is a kind of tracker of statistics for an underlying gauge metrics, and you reset it (by clearing history) whenever you want. Is that correct?

The initial idea is to have a layer on top of the raw metrics that averages it over X samples. Precision is not important since it's only used for UI display. What's more important is to be able to eyeball the metrics quickly (hence the window average). Because this is for quick eyeballing investigation, it's a "nice to have" to be able to disable recording.

That's for the goal/idea I have in mind.

Keenuts avatar Jan 17 '24 13:01 Keenuts

It is a kind of tracker of statistics for an underlying gauge metrics, and you reset it (by clearing history) whenever you want. Is that correct?

Addition to Nathan's comment. The use case is to tracks a set of gauge metrics that we want to see on screen, not necessary the ones we want to record to a profiling run (e.g. cpu time spent on input processing). The samples are $2^{-t_{elapsed}}$ weighted to make sure we are showing recent information (also this formulation allows constant time update). Please note this is not the last X samples since interval between sample can vary from $<0.1ms$ to $100^+ ms$ . The clearing of history is somewhat unnecessary, since old data's weight will decay to almost zero in a few seconds, it's mainly done to since if we know parameter has changed, immediate update is better than waiting a few seconds.

hysw avatar Jan 17 '24 19:01 hysw

Adding Angela & Ran since they have reviewed #377 .

hysw avatar Jan 17 '24 20:01 hysw

Could you please add a quick summary of what problem the PR solves? I am not extremely familiar with the benchmark, and I have not touched BigWheels for a few months.

This gives a readable statistics for cpu submission time / gpu work duration / etc. i.e. the leading digit does not jump around every frame. This version is due to suggestion of @Keenuts 's suggestion on have it in the metric framework. The stand alone version is #377 .

It is a kind of tracker of statistics for an underlying gauge metrics, and you reset it (by clearing history) whenever you want. Is that correct?

The initial idea is to have a layer on top of the raw metrics that averages it over X samples. Precision is not important since it's only used for UI display. What's more important is to be able to eyeball the metrics quickly (hence the window average). Because this is for quick eyeballing investigation, it's a "nice to have" to be able to disable recording.

That's for the goal/idea I have in mind.

Ok, thx. Makes sense.

mdagois avatar Jan 18 '24 01:01 mdagois

LGTM from a metrics perspective. I'll defer approval to people that own the project.

mdagois avatar Jan 18 '24 02:01 mdagois

Personally though, I find the metrics framework more confusing with these changes, and not sure if the complexity is justified.

My preference is keeping it simple and separate, however Nathan have some good point, so I also made this version to see if integration makes sense.

hysw avatar Jan 18 '24 18:01 hysw