questdb icon indicating copy to clipboard operation
questdb copied to clipboard

chore(ilp): optimize parser code on hot path

Open puzpuzpuz opened this issue 2 years ago • 2 comments

WIP

Intermediate TSBS benchmarks on local machine.

Before:

puzpuzpuz@apechkurov-laptop:~/projects/tsbs/bin$ GOMAXPROCS=2 ./tsbs_load_questdb --file /tmp/data --workers 1
time,per. metric/s,metric total,overall metric/s,per. row/s,row total,overall row/s
1655103689,2429550.36,2.430000E+07,2429550.36,242955.04,2.430000E+06,242955.04
1655103699,2499935.04,4.930000E+07,2464739.90,249993.50,4.930000E+06,246473.99
1655103709,2440140.53,7.370000E+07,2456541.00,244014.05,7.370000E+06,245654.10
1655103719,2389846.52,9.760000E+07,2439867.22,238984.65,9.760000E+06,243986.72
^C
puzpuzpuz@apechkurov-laptop:~/projects/tsbs/bin$ GOMAXPROCS=2 ./tsbs_load_questdb --file /tmp/data --workers 1
time,per. metric/s,metric total,overall metric/s,per. row/s,row total,overall row/s
1655103798,3679632.82,3.680000E+07,3679632.82,367963.28,3.680000E+06,367963.28
1655103808,3669999.70,7.350000E+07,3674816.50,366999.97,7.350000E+06,367481.65
1655103818,3610246.50,1.096000E+08,3653294.86,361024.65,1.096000E+07,365329.49
1655103828,3739876.95,1.470000E+08,3674940.75,373987.70,1.470000E+07,367494.07
^C
puzpuzpuz@apechkurov-laptop:~/projects/tsbs/bin$ GOMAXPROCS=2 ./tsbs_load_questdb --file /tmp/data --workers 1
time,per. metric/s,metric total,overall metric/s,per. row/s,row total,overall row/s
1655104001,3329710.60,3.330000E+07,3329710.60,332971.06,3.330000E+06,332971.06
1655104011,3180113.59,6.510000E+07,3254916.68,318011.36,6.510000E+06,325491.67
1655104021,2439851.58,8.950000E+07,2983221.93,243985.16,8.950000E+06,298322.19
1655104031,2470115.39,1.142000E+08,2854953.38,247011.54,1.142000E+07,285495.34
1655104041,2449927.56,1.387000E+08,2773947.36,244992.76,1.387000E+07,277394.74
1655104051,2429975.33,1.630000E+08,2716619.11,242997.53,1.630000E+07,271661.91
^C

After:

puzpuzpuz@apechkurov-laptop:~/projects/tsbs/bin$ GOMAXPROCS=2 ./tsbs_load_questdb --file /tmp/data --workers 1
time,per. metric/s,metric total,overall metric/s,per. row/s,row total,overall row/s
1655104218,2289775.51,2.290000E+07,2289775.51,228977.55,2.290000E+06,228977.55
1655104228,2340160.38,4.630000E+07,2314965.85,234016.04,4.630000E+06,231496.58
1655104238,2619890.98,7.250000E+07,2416609.38,261989.10,7.250000E+06,241660.94
1655104248,2680036.03,9.930000E+07,2482464.21,268003.60,9.930000E+06,248246.42
1655104258,2580051.56,1.251000E+08,2501981.14,258005.16,1.251000E+07,250198.11
1655104268,2319948.45,1.483000E+08,2471641.99,231994.85,1.483000E+07,247164.20
^C
puzpuzpuz@apechkurov-laptop:~/projects/tsbs/bin$ GOMAXPROCS=2 ./tsbs_load_questdb --file /tmp/data --workers 1
time,per. metric/s,metric total,overall metric/s,per. row/s,row total,overall row/s
1655104344,2959728.73,2.960000E+07,2959728.73,295972.87,2.960000E+06,295972.87
1655104354,2779998.84,5.740000E+07,2869867.88,277999.88,5.740000E+06,286986.79
1655104364,2790012.71,8.530000E+07,2843250.39,279001.27,8.530000E+06,284325.04
1655104374,3279948.63,1.181000E+08,2952423.84,327994.86,1.181000E+07,295242.38
1655104384,2990232.14,1.480000E+08,2959984.88,299023.21,1.480000E+07,295998.49
1655104394,2779797.71,1.758000E+08,2929951.99,277979.77,1.758000E+07,292995.20
^C
puzpuzpuz@apechkurov-laptop:~/projects/tsbs/bin$ GOMAXPROCS=2 ./tsbs_load_questdb --file /tmp/data --workers 1
time,per. metric/s,metric total,overall metric/s,per. row/s,row total,overall row/s
1655104453,2099837.06,2.100000E+07,2099837.06,209983.71,2.100000E+06,209983.71
1655104463,1990007.35,4.090000E+07,2044924.44,199000.74,4.090000E+06,204492.44
1655104473,2430062.96,6.520000E+07,2173298.56,243006.30,6.520000E+06,217329.86
1655104483,2359869.49,8.880000E+07,2219942.67,235986.95,8.880000E+06,221994.27
1655104493,1969981.45,1.085000E+08,2169951.08,196998.14,1.085000E+07,216995.11
^C

puzpuzpuz avatar Jun 10 '22 19:06 puzpuzpuz

[PR Coverage check]

:heart_eyes: pass : 22 / 23 (95.65%)

file detail

path covered line new line coverage
:large_blue_circle: io/questdb/cutlass/line/tcp/LineTcpParser.java 22 23 95.65%

ideoma avatar Jun 14 '22 14:06 ideoma

I'll keep this PR pending for some time. The thing is that there is a source of jitter in the scenario under test and I'd like to understand and fix it first. Otherwise, it's very problematic to judge on any optimizations.

puzpuzpuz avatar Jun 28 '22 07:06 puzpuzpuz

Closing this one since it's been handing in the air for a while. Hope to return back to parser optimization one day.

puzpuzpuz avatar Sep 02 '22 08:09 puzpuzpuz