MediaStreamTrack Statistics

[henbos]

WebKittens

@youennf @jyavenard

Title of the spec

Media Capture and Streams Extensions, section "MediaStreamTrack Statistics"

URL to the spec

https://w3c.github.io/mediacapture-extensions/#mediastreamtrack-statistics

URL to the spec's repository

https://github.com/w3c/mediacapture-extensions

Issue Tracker URL

No response

Explainer URL

No response

TAG Design Review URL

No response

Mozilla standards-positions issue URL

https://github.com/mozilla/standards-positions/issues/895

WebKit Bugzilla URL

No response

Radar URL

No response

Description

Video conferencing web developers (Google Meet) have asked for an API to know the actual camera frame rate as well as an ability to get a signal from the stack if frames are being dropped somewhere in the pipeline.

The problem with MediaStreamTrack.getSettings().frameRate is that it only provides you with the camera's configured frame rate, say 30. In reality, it is common for the actual frame rate to vary such as due to low lighting conditions. In rare circumstances, frames could even get dropped by the user agent, such as if the system is under heavy load or buffers are not being released on time for some reason.

The track stats API provides valuable insights. Here are real world scenarios that have been observed, but have been hard to investigate without this API:

• A USB hub between the camera and the PC reduced the camera fps (track.stats.totalFrames/s) in half.
• In screensharing of static content, an app may only want to encode frames when the screen has changed (e.g. change presentation slide). An app that does this may find that a frame is discarded due to the frame decimation algorithm (++track.stats.discardedFrames) and needs to send to produce another frame (else the next presentation slide is never encoded and sent to the other participants).
• If the system is under heavy load or there are other issues with the stack, frames could be dropped for other reasons (track.stats.totalFrames - track.stats.deliveredFrames - track.stats.discardedFrames > 0). This is rare but it deserves to be flagged when it happens.

Having an API can help notify the user, troubleshoot (e.g. USB hub...) or provide A/B diagnostics when a web app is launching new features. The motivation is similar to the RTCPeerConnection.getStats() API, but there are MediaStreamTrack use cases that don't depend on WebRTC so it makes more sense to have the API on the track.

The metrics are expressed as total counters as opposed to frame rates because this allows the app to calculate average frame rates over any interval (by polling twice and looking at delta frames / delta seconds) or to provide total number of frames dropped or discarded for the entire capture session. Some apps want rates for every second, others every 10 seconds, and other may just care about the session total. In the case of discarded/dropped frames there are use cases where total counters are preferred, or else the app has to poll excessively in order not to "miss" any time a drop happened.

Other information

At the time of writing, the track.stats API is only defined for video tracks. I plan to submit a PR that also adds audio stats which will have different metrics definition and a different interface, but the API shape will follow the same pattern, i.e. the app should be able to call track.stats on an audio track in the future as well.

Edit audio track stats are now also part of the spec, see comments below

[henbos]

For background, the last time this API was discussed in the WebRTC Working Group was at TPAC, where we decided to make the API a synchronous getter instead of a promise-based API that resolves with the result: recording with timestamp, slides

[henbos]

Chrome has already shipped video track stats, I am planning to do an Intent to Ship for the audio track stats as well. Shall I reuse this "standards-position" to cover both audio and video track stats or should I file a new one for the audio track stats case, @youennf ?

The update is that we've extended this API to also include getUserMedia-sourced audio tracks: https://w3c.github.io/mediacapture-extensions/#the-mediastreamtrackaudiostats-interface

The audio metrics defined include:

• Frame counters! Just like in the video case, you can calculate dropped frames ("dropped = total - delivered"), however in the audio case there is no such thing as discardedFrames because the UA should try to deliver all frames.
• Audio frames may be dropped by the device (e.g. poor bluetooth connection), by the OS (e.g. skipped timpestamps from OS callback) or possibly inside the UA.
• track.stats.latency measures the microphone input delay. We also have a PR that adds min/max/average latency since resetLatency() is called which has not merged at the time of writing but it should be included in the spec since it was approved by the working group in the meeting yesterday.