Flawed benchmark interpretation?

[somebee]

Hi. From reading your findings over at the auth0 blog, I'm wondering if you might be interpreting several of the results incorrectly. As you state:

Layout/Paint. This graph shows the time spent by the browser doing re-layout operations (i.e. creating an internal representation of the DOM tree after changes). I find it surprising that cito.js is the slowest in this case. Subjective performance would tell otherwise, as both cito.js and Incremental DOM feel quite snappy when interacting directly with the browser.

The less time each framework has spent doing layout/paint does not really mean that it is faster. I could make a framework that was incredibly slow, resulting in very little time being spent in layout/paint, simply because it spends the time doing other things? Also, the more rendering loops a framework manages to push during the benchmark (higher fps) - the more time will be spent in layout/paint (if the frameworks invalidate/update the same amount of dom nodes on each render).

I guess a better benchmark would be to run each implementation for a specific amount of runs - and then compare the time spent doing this (this will also give a fairer result of how much time is spent in gc etc, since all implementations have done the same amount of work). I think the results would be radically different :)