Benchmark with criterion

[ndmitchell]

Specifically should benchmark operations against utf8-string, bytestring, text and vector. Ideally would be best-of-breed on all of those, and quite an easy criterion to write.

[vincenthz]

my initial aim is not to be faster than all of this, as this is quite a stretch, but hopefully not being too far. There's lot of room for improvement everywhere including some obvious low hanging fruits, and clearly it wouldn't be fair to expect to be compared to optimised and solid libraries.

Clearly quantifying the difference is the first goal; then we'll be able to prioritise the efforts to bring it up-to-par or better.

[ndmitchell]

Yeah, I meant this ticket to just write the benchmark, which will tell us where we are behind and what the difference is.

[vincenthz]

Tiny first benchmark that is really encouraging :-)

doing a break on a String with foundation: time 51.28 μs (49.78 μs .. 52.97 μs)

doing a break on a String with text: time 55.12 μs (54.49 μs .. 55.90 μs)

edit: no optimisation. slower by a x10 margin otherwise.

[vincenthz]

FYI with optimisation on (-O), and a set of hardcore -- violate-all-the-abstraction-layers -- optimisations, foundation is 3.5x faster doing a break on ascii character:

foundation: time 30.94 ns (30.55 ns .. 31.35 ns)

text: time 106.6 ns (105.4 ns .. 107.8 ns)

[vincenthz]

The last numbers (and I'm stopping here for now), show only 2ns slower to start (which is easy to explain since foundation handle 2 types of backend String), but otherwise couple of nanoseconds faster to find in the middle of the string, and iterate the whole string (not found):

benchmarking break/#english-start/foundation
time                 17.26 ns   (17.08 ns .. 17.43 ns)
benchmarking break/#english-start/text
time                 15.30 ns   (15.11 ns .. 15.53 ns)

benchmarking break/#english-middle/foundation
time                 101.3 ns   (99.08 ns .. 103.5 ns)
benchmarking break/#english-middle/text
time                 108.4 ns   (105.9 ns .. 111.1 ns)

benchmarking break/#english-notfound/foundation
time                 376.0 ns   (368.8 ns .. 383.6 ns)
benchmarking break/#english-notfound/text
time                 384.7 ns   (380.4 ns .. 389.6 ns)

[ndmitchell]

Nice! Can you try similar benchmarks with vector and bytestring?

[vincenthz]

Any simple bench idea for those 2 ?

[ndmitchell]

Exactly the same break test, just on ASCII.

[vincenthz]

For bytestring, replacing my initial UArray breakElem with the new UArray splitElem that I made for String, I get the following (on a really quick benchmark):

benchmarking break/#word8-start/foundation
time                 15.46 ns   (15.23 ns .. 15.73 ns)
benchmarking break/#word8-start/bytestring
time                 15.04 ns   (14.74 ns .. 15.31 ns)

benchmarking break/#word8-middle/foundation
time                 107.8 ns   (106.3 ns .. 109.2 ns)
benchmarking break/#word8-middle/bytestring
time                 140.8 ns   (136.6 ns .. 144.3 ns)

benchmarking break/#word8-end/foundation
time                 204.2 ns   (199.6 ns .. 208.6 ns)
benchmarking break/#word8-end/bytestring
time                 264.6 ns   (259.9 ns .. 269.3 ns)

[vincenthz]

that's benchmarking a UArray Word8 against a ByteString doing a break at difference places (beggining, middle, and end) in a 256 elements array

[vincenthz]

insanely slow for vector. exact same thing as above instead using vector unboxed to replace the bytestring:

benchmarking break/#word8-start/foundation
time                 15.46 ns   (15.23 ns .. 15.70 ns)
benchmarking break/#word8-start/vector-unboxed
time                 18.45 ns   (18.21 ns .. 18.70 ns)

benchmarking break/#word8-middle/foundation
time                 110.4 ns   (109.0 ns .. 112.2 ns)
benchmarking break/#word8-middle/vector-unboxed
time                 1.211 μs   (1.188 μs .. 1.233 μs)

benchmarking break/#word8-end/foundation
time                 202.9 ns   (198.0 ns .. 207.8 ns)
benchmarking break/#word8-end/vector-unboxed
time                 2.522 μs   (2.478 μs .. 2.565 μs)

[ndmitchell]

Very cool - perhaps you should also benchmark foundation break on UArray vs String.

[vincenthz]

it really should go almost at the same speed now (provided ascii Char)

[ndmitchell]

Statements about performance is not always the same as actual performance :) - would be good to confirm (and very nice if it was).

[vincenthz]

confirmed (sometimes it pull ahead of uvec too):

benchmarking break/string
time                 507.0 ns   (499.7 ns .. 513.9 ns)
                     0.998 R²   (0.996 R² .. 0.999 R²)
mean                 504.3 ns   (496.4 ns .. 512.5 ns)
std dev              27.60 ns   (22.99 ns .. 33.41 ns)
variance introduced by outliers: 72% (severely inflated)

benchmarking break/uvec
time                 501.6 ns   (493.1 ns .. 509.8 ns)
                     0.997 R²   (0.996 R² .. 0.998 R²)
mean                 500.8 ns   (493.4 ns .. 510.9 ns)
std dev              30.51 ns   (25.53 ns .. 40.45 ns)
variance introduced by outliers: 76% (severely inflated)

[ndmitchell]

Very awesome. Once this is integrated into the CI scripts etc I suggest closing this ticket, and ideally everything that gets tested should be benchmarked as well.