SuperString icon indicating copy to clipboard operation
SuperString copied to clipboard

C++ benchmark is inaccurate

Open zig-for opened this issue 7 years ago • 7 comments

https://github.com/btwael/SuperString/blob/master/test/withStd.cc#L26 this line always copies from the current character to the very end of the file (rather than just to the end of the line). Also, the last line is skipped. The memory used should be much less than 450MB.

zig-for avatar Apr 12 '18 07:04 zig-for

I think this should be reopened as the benchmark are indeed still inaccurate. I actually couldn't replicate the results claimed in the readme but I'm not sure if my setup is completely correct. I had to add the for(auto _ : state) in the bench_compare.cc file to get any statistic from the benchmark library, and the execution times where about 60x slower for the SuperString implementation :/ The number of CPU iteration was smaller though

fcortes avatar Apr 12 '18 19:04 fcortes

Please reopen - https://raw.githubusercontent.com/btwael/SuperString/master/documentation/img/text_line_split_bench.png is wrong.

zig-for avatar Apr 12 '18 20:04 zig-for

In a last test, std::string consumed 11MB, while supersrring consumed 10MB. for the moment, don't use superstring with small amount of data. Rope structure are more performant with heavy text manipulation.

Also the library may have better benchmark result in 32bit than 64bit machine

btwael avatar Apr 12 '18 20:04 btwael

I have to admit that the api difference between std::string::substr(startIndex, length) and SuperString::substring(startIndex, endInex) tricked me, and many reviews from reddit and hacker news show how code quality can be improved, you can also remark that there is no big difference between std::string and SuperString (in general usage, may steel useful for big manipulation).

The ideas behind SuperString are amazing, but the usage of a lot of pointers (8 bytes in x64, that's 8 ASCII character) kills the advantage of rope data structures.

So I will try to implement additional data representation for short manipulation (e.g. if you the result of substring (number of character) is less than the substring structure (pointer to the original string, startIndex, endIndex)) just copy the data.

Thank you for being interested, and sorry because my bar plot was tricky and inaccurate, I will try to fix that as soon as possible.

I want also to refers this blog post that inspired me to create SuperString (the blog post is very accurate)

btwael avatar Apr 12 '18 20:04 btwael

Try doing an operation with a lot of concatenations - it may show off the power of your library better.

zig-for avatar Apr 14 '18 09:04 zig-for

The misleading benchmarks should be removed until they are replaced with more accurate ones.

ghost avatar Apr 17 '18 20:04 ghost

The misleading benchmarks are removed, the development is focused on making it compatible with std::string so we can test in real-world applications.

btwael avatar Apr 18 '18 19:04 btwael