sitespeedio
ContentfulSpeedIndex (CSI) is currently problematic because it punishes staying on the last frame if the last frame is not the most contentful
The CSI algorithm implemented in visualmetrics.py currently does the following:
- For each frame, calculate a contentfulness score (based on edge detection and counting "edge" pixels).
- Find the maximum contentfulness across all frames.
- For each frame, calculate the percentage by which the frame differs from the maximum contentfulness.
- Add up the differences weighted by time.
This works ok if the last frame is the most contentful frame. However, if maximum contentfulness was reached before the final state, then the final state gets puninshed: The longer the video stays in the final state, the higher (worse) the score becomes, because it's not at its most contentful during that time. The duration of the final state in the captured videos is somewhat arbitrary. We rely somewhat on the assumption that some extra video seconds in the final state don't affect the score - but for CSI they do.
I have an example of a Fenix and a Chrome recording where Chrome puts content on the screen 200ms before Fenix. But the CSI score for Chrome is worse (higher), because it briefly shows some extra content (a gray line at the right edge of the viewport) that then goes away again, and this extra content punishes all remaining frames that don't have it.
Proposal: Rather than computing the maximum contentfulness, I think we should compare all frames to the "final" contentfulness. If a frame in the middle has more contentfulness than the final frame, then that frame should be treated as if it had the same contentfulness as the final frame. This rewards putting content on the screen early, and it doesn't punish staying in the final state longer.
So there would be no penalty for having "too much" contentfulness (relative to final) during the pageload? Just to verify my understanding, not a real concern: Could one not render random high contentful graphics during pageload and then display the final image to "game" a higher CSI?
I don't know if punishing "too much" contentfulness would be good or bad. We might want to experiment with both. Do we want to reward "any kind of content" on the screen or only "the right (amount of) content"? And yes, CSI is easily gameable just by putting noise on the screen earlier. I think the thinking behind CSI was that any kind of content is good to have on the screen early, even if it doesn't match the final content.
