Speed Weirdness...

[HamptonMakes]

First of all, thank you all for doing this project! This has given us all a great metric to improve our speeds over in Sass-land. Great to have a reasonable benchmark that we can all use to track both our own performance changes, and compare those to similar solutions.

Before we released node-sass 3.4.0, we ran these tests and it showed libsass having about a 1.2x (on average) advantage. When I saw the updates on the main page, I was surprised to see us being 1.2x slower after this release. I pulled the repo back down and ran them again, getting the following:

libsass:   93 ms   (1.2 times faster)
Rework:    105 ms  (1.0 times faster)
PostCSS:   108 ms
Less:      286 ms  (2.6 times slower)
Stylus:    309 ms  (2.9 times slower)
Stylecow:  384 ms  (3.5 times slower)
Ruby Sass: 2488 ms (23.0 times slower)

PostCSS:   75 ms
libsass:   83 ms   (1.1 times slower)
Rework:    94 ms   (1.2 times slower)
Less:      228 ms  (3.0 times slower)
Stylus:    277 ms  (3.7 times slower)
Stylecow:  324 ms  (4.3 times slower)
Ruby Sass: 1752 ms (23.3 times slower)

libsass:   83 ms   (1.1 times faster)
PostCSS:   93 ms
Rework:    126 ms  (1.4 times slower)
Less:      247 ms  (2.7 times slower)
Stylus:    296 ms  (3.2 times slower)
Stylecow:  331 ms  (3.5 times slower)
Ruby Sass: 1907 ms (20.5 times slower)


PostCSS:   78 ms
libsass:   80 ms   (1.0 times slower)
Rework:    95 ms   (1.2 times slower)
Less:      234 ms  (3.0 times slower)
Stylus:    242 ms  (3.1 times slower)
Stylecow:  332 ms  (4.2 times slower)
Ruby Sass: 1869 ms (23.8 times slower)

Also, you're last Travis-CI build of the project (https://travis-ci.org/postcss/benchmark/builds/87379668) has the following:

libsass:   87 ms   (1.2 times faster)
PostCSS:   102 ms
Rework:    137 ms  (1.3 times slower)
Less:      217 ms  (2.1 times slower)
Stylus:    268 ms  (2.6 times slower)
Stylecow:  334 ms  (3.3 times slower)
Ruby Sass: 1442 ms (14.1 times slower)

So, definitely seems like we either need to do more samples or mix up the way they are run. Because, Rework, PostCSS, and Sass are all pretty neck-and-neck at the moment.

I did try and increase the number of dynamic lines (from 100 to 1000 iterations) for all the processors and that definitely went way in libsass' favour.

libsass:  201 ms  (1.5 times faster)
Rework:   246 ms  (1.2 times faster)
PostCSS:  303 ms
Less:     587 ms  (1.9 times slower)
Stylus:   706 ms  (2.3 times slower)
Stylecow: 1411 ms (4.7 times slower)

libsass:  199 ms  (1.5 times faster)
Rework:   243 ms  (1.2 times faster)
PostCSS:  299 ms
Less:     585 ms  (2.0 times slower)
Stylus:   723 ms  (2.4 times slower)
Stylecow: 1410 ms (4.7 times slower)

But clearly, someone from a "competing" technology can't just change the rules of the benchmark that happen to benefit them, so I'm not going to PR going with the larger files.

I couldn't figure out how to increase the number of runs (to lower the variations), because I think that's hidden in "gulp-benchmark" somewhere, and I'm not an experienced gulp person.

I know the point of this project isn't to be always fighting over small number changes, but we should find a way to get better consistency.

Again, thanks for this project! <3

[ai]

Libsass team made really great work in perfomance and I think that libsass right now is faster. But you right, we have a problem with variations.

I have less variations. But before benchmark and reboot system and do not run any other applications.

This is why I think the problem can be in hardware. I have slow CPU and good RAM.

So maybe we can use Travis result as some ideal reference?

[HamptonMakes]

Even travis seems to disagree and flip-flop us!

libsass wins by 1.2 https://travis-ci.org/postcss/benchmark/builds/87379668 postcss wins by 1.1 https://travis-ci.org/postcss/benchmark/builds/87299476

And no change in code!

[ai]

Closing since there is no way to fix it