For all resources, add timestamp of when it was available to the browser

[bluesmoon]

What happens

There are cases when a resource fetched over the network has completed downloading, and is available to the browser, but the browser is too busy to read it and parse it. In these cases, the responseStart timestamp is set after the browser becomes free to read the file off its buffer. This incorrectly points to a network problem.

Example

As an example, using WebPageTest, we measured the loading of a particular website, and also captured the network PCAP file of the data transfer. The PCAP file showed that the file had been fetched (all packets arrived at destination) in under 200ms, however the browser reported responseStart at the 1.2second mark. WPT showed that the browser was at 100% CPU utilization and 0% bandwidth utilization during these 1.2 seconds. This indicated that while the file was available locally in some buffer, the browser was too busy to read it, and incorrectly put the blame on the network.

What we'd like to have

1. responseStart should stamp when bytes of the response were available, not when the browser was ready to parse them
2. responseEnd should stamp when the response was completely available, not when the browser finished parsing
3. Add a new timestamp called loadComplete or resourceLoaded or equivalent, indicating when the browser actually finished handling the resource in a way appropriate for that resource (which might include inserting into the DOM, parsing, executing, etc.)

I believe 1 & 2 are mostly in line with the spec, we should make it explicit that the browser being unable to load, parse and execute an available resource should not delay the responseStart and responseEnd timestamps, and browsers need to follow this.

Source Discussion

This came up as part of the W3C discussion at the Velocity Conference in Santa Clara, CA on 2016-06-22T19:00PDT.

Members in attendance during discussion: (please update list in case I've missed/misspelt anyone)

• Ryosuke Niwa
• Ilya Grigorik
• Philip Tellis

[igrigorik]

Hmm, proposed response{Start/End} is tricky...

1. It's perfectly plausible that response bytes make it to a kernel buffer and sit there for a while before being consumed by the browser's network stack.
2. Most modern browsers (all?) have a separate network process.. response bytes can sit there for a while until the renderer is ready to consume them. However, I'm not sure if we can expose this in a compatible way across all the browsers, because some may rely on OS-provided libraries to handle this (@toddreifsteck, how does Edge handle this today.. you guys rely on WinINet, right?)

Re, loadcomplete, I think we'd just want to attach it to the load event. In fact, we could probably just move loadEvent{Start,End} from NavTiming into RT..

[bluesmoon]

As long as we get something that lets us split between network time and client time, that is sufficient.

[bmaurer]

Regarding responseStart and responseEnd I think that the spec should request that the browser get as close as possible to the time when the kernel got the relevant bytes. There's always going to be portability differences here -- Whatever thread is used to measure this could be busy, waiting for a lock, etc. I agre with Ilya that the load event can assist in measuring the load time. But I actually think the events could be useful:

• Having a start vs end event (especially for scripts) is useful. Namely, right not it's impossible to measure the time spent compiling a script tag or to parse a stylesheet.
• Some browsers seem to put their onload event into a queue rather than execute them immediately (I think chrome does this with stylesheets). This means that other events in the main thread can delay the load time.
• From a tooling perspective I think it will be easier to write generic tools if there's a load event (since tooling won't need to insert event handlers on scripts/styles). Also this minimizes the amount of code one needs to write for instrumentation (much of which needs to be inlined so that it can measure the earliest scripts).