Use simdjson for binary json construction for improved performance

[swalrus1]

Use simdjson as a backend for conversion from string to binary json.

benchmarks show 220% performance improvement of binary json construction

-------------------------------------------------------------
Benchmark                   Time             CPU   Iterations
-------------------------------------------------------------
BenchWriteRapidJson      46687 ns        46687 ns       151833

-------------------------------------------------------------
Benchmark                   Time             CPU   Iterations
-------------------------------------------------------------
BenchWriteSimdJson      21247 ns        21247 ns       330340

[github-actions[bot]]

:white_circle: 2024-10-15 15:20:45 UTC Pre-commit check linux-x86_64-relwithdebinfo for 17a887f6acfc8aafa752088de6314e6494a5923a has started. :white_circle: 2024-10-15 15:20:55 UTC Artifacts will be uploaded here :white_circle: 2024-10-15 15:24:41 UTC ya make is running... :red_circle: 2024-10-15 15:47:54 UTC Build failed, see the logs.

[github-actions[bot]]

:white_circle: 2024-10-15 15:21:01 UTC Pre-commit check linux-x86_64-release-asan for 17a887f6acfc8aafa752088de6314e6494a5923a has started. :white_circle: 2024-10-15 15:21:11 UTC Artifacts will be uploaded here :white_circle: 2024-10-15 15:24:58 UTC ya make is running... :red_circle: 2024-10-15 15:47:28 UTC Build failed, see the logs.

[github-actions[bot]]

:white_circle: 2024-10-15 16:13:01 UTC Pre-commit check linux-x86_64-relwithdebinfo for 9a25bc230c00632450690f03497872dd8919dc97 has started. :white_circle: 2024-10-15 16:13:12 UTC Artifacts will be uploaded here :white_circle: 2024-10-15 16:17:00 UTC ya make is running... :red_circle: 2024-10-15 17:39:41 UTC Too many tests failed, NOT going to retry :red_circle: 2024-10-15 17:40:57 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
79619	64417	0	828	14267	107

:green_circle: 2024-10-15 17:48:34 UTC Build successful. :green_circle: 2024-10-15 17:48:58 UTC ydbd size 2.8 GiB changed* by +9.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: 3ab7dd796b22380f98b56f8c7fdb1cdce512cec3	merge: 9a25bc230c00632450690f03497872dd8919dc97	diff	diff %
ydbd size	3 010 665 048 Bytes	3 010 674 936 Bytes	+9.7 KiB	+0.000%
ydbd stripped size	477 552 856 Bytes	477 555 032 Bytes	+2.1 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

[github-actions[bot]]

:white_circle: 2024-10-15 16:13:03 UTC Pre-commit check linux-x86_64-release-asan for 9a25bc230c00632450690f03497872dd8919dc97 has started. :white_circle: 2024-10-15 16:13:14 UTC Artifacts will be uploaded here :white_circle: 2024-10-15 16:16:58 UTC ya make is running... :yellow_circle: 2024-10-15 18:07:57 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
16710	15809	0	318	543	40

:green_circle: 2024-10-15 18:09:25 UTC Build successful. :green_circle: 2024-10-15 18:10:00 UTC ydbd size 5.7 GiB changed* by +17.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: 3ab7dd796b22380f98b56f8c7fdb1cdce512cec3	merge: 9a25bc230c00632450690f03497872dd8919dc97	diff	diff %
ydbd size	6 099 382 840 Bytes	6 099 401 072 Bytes	+17.8 KiB	+0.000%
ydbd stripped size	1 521 054 992 Bytes	1 521 060 176 Bytes	+5.1 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

[github-actions[bot]]

:white_circle: 2024-10-16 10:47:27 UTC Pre-commit check linux-x86_64-release-asan for e4cd4921b92265858913368d1b8e26f6f34da27c has started. :white_circle: 2024-10-16 10:47:38 UTC Artifacts will be uploaded here :black_circle: 2024-10-16 10:48:32 UTC Check cancelled

[github-actions[bot]]

:white_circle: 2024-10-16 10:51:06 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4fd7248130efa40a3b473b3839230a6b2a44ecf0 has started. :white_circle: 2024-10-16 10:51:38 UTC Artifacts will be uploaded here :black_circle: 2024-10-16 10:54:49 UTC Check cancelled

[github-actions[bot]]

:white_circle: 2024-10-16 10:53:15 UTC Pre-commit check linux-x86_64-release-asan for 4fd7248130efa40a3b473b3839230a6b2a44ecf0 has started. :white_circle: 2024-10-16 10:53:26 UTC Artifacts will be uploaded here :black_circle: 2024-10-16 10:54:50 UTC Check cancelled

[github-actions[bot]]

:white_circle: 2024-10-16 10:57:48 UTC Pre-commit check linux-x86_64-release-asan for aac36963c922ec6f48459605568c017a7fee6694 has started. :white_circle: 2024-10-16 10:58:00 UTC Artifacts will be uploaded here :white_circle: 2024-10-16 11:01:44 UTC ya make is running... :red_circle: 2024-10-16 11:07:23 UTC Build failed, see the logs. Also see fail summary

[github-actions[bot]]

:white_circle: 2024-10-16 10:58:01 UTC Pre-commit check linux-x86_64-relwithdebinfo for aac36963c922ec6f48459605568c017a7fee6694 has started. :white_circle: 2024-10-16 10:58:44 UTC Artifacts will be uploaded here :white_circle: 2024-10-16 11:02:59 UTC ya make is running... :red_circle: 2024-10-16 11:09:20 UTC Build failed, see the logs. Also see fail summary

[github-actions[bot]]

:white_circle: 2024-10-16 11:31:53 UTC Pre-commit check linux-x86_64-release-asan for d98dfdd9e0fb0d1cde90c76f4c4949e5992cd523 has started. :white_circle: 2024-10-16 11:32:27 UTC Artifacts will be uploaded here :white_circle: 2024-10-16 11:36:38 UTC ya make is running... :red_circle: 2024-10-16 11:43:22 UTC Build failed, see the logs. Also see fail summary

[github-actions[bot]]

:white_circle: 2024-10-16 11:32:45 UTC Pre-commit check linux-x86_64-relwithdebinfo for d98dfdd9e0fb0d1cde90c76f4c4949e5992cd523 has started. :white_circle: 2024-10-16 11:32:57 UTC Artifacts will be uploaded here :white_circle: 2024-10-16 11:36:45 UTC ya make is running... :red_circle: 2024-10-16 11:40:04 UTC Build failed, see the logs. Also see fail summary

[github-actions[bot]]

:white_circle: 2024-10-16 12:43:11 UTC Pre-commit check linux-x86_64-release-asan for a4cdb4e43c25269329f3204ec3b6dd423333b50b has started. :white_circle: 2024-10-16 12:43:46 UTC Artifacts will be uploaded here :white_circle: 2024-10-16 12:48:04 UTC ya make is running... :red_circle: 2024-10-16 12:55:28 UTC Build failed, see the logs. Also see fail summary

[github-actions[bot]]

:white_circle: 2024-10-16 12:45:10 UTC Pre-commit check linux-x86_64-relwithdebinfo for a4cdb4e43c25269329f3204ec3b6dd423333b50b has started. :white_circle: 2024-10-16 12:45:21 UTC Artifacts will be uploaded here :white_circle: 2024-10-16 12:49:05 UTC ya make is running... :red_circle: 2024-10-16 13:09:59 UTC Build failed, see the logs. Also see fail summary

[github-actions[bot]]

:white_circle: 2024-10-16 13:56:38 UTC Pre-commit check linux-x86_64-relwithdebinfo for e22315fe8b71e88b95ae647aaff0c7d785a2e4ff has started. :white_circle: 2024-10-16 13:57:09 UTC Artifacts will be uploaded here :white_circle: 2024-10-16 14:01:24 UTC ya make is running... :black_circle: 2024-10-16 14:42:36 UTC Check cancelled

[github-actions[bot]]

:white_circle: 2024-10-16 13:59:58 UTC Pre-commit check linux-x86_64-release-asan for e22315fe8b71e88b95ae647aaff0c7d785a2e4ff has started. :white_circle: 2024-10-16 14:00:30 UTC Artifacts will be uploaded here :white_circle: 2024-10-16 14:04:58 UTC ya make is running... :black_circle: 2024-10-16 14:42:47 UTC Check cancelled

[github-actions[bot]]

:white_circle: 2024-10-16 15:11:24 UTC Pre-commit check linux-x86_64-relwithdebinfo for 0c83a9a26cafd16f60c9fa9e734bc155f96aec9e has started. :white_circle: 2024-10-16 15:12:16 UTC Artifacts will be uploaded here :white_circle: 2024-10-16 15:16:37 UTC ya make is running... :yellow_circle: 2024-10-16 16:24:45 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
79602	65616	0	12	13871	103

:white_circle: 2024-10-16 16:32:46 UTC ya make is running... (failed tests rerun, try 2) :yellow_circle: 2024-10-16 16:44:40 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
114 (only retried tests)	12	0	2	1	99

:white_circle: 2024-10-16 16:44:48 UTC ya make is running... (failed tests rerun, try 3) :red_circle: 2024-10-16 16:56:06 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
101 (only retried tests)	1	0	2	0	98

:green_circle: 2024-10-16 16:56:13 UTC Build successful. :green_circle: 2024-10-16 16:56:35 UTC ydbd size 2.8 GiB changed* by +62.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: 8739de41d2d04556ceac159aac92929c16b14a57	merge: 0c83a9a26cafd16f60c9fa9e734bc155f96aec9e	diff	diff %
ydbd size	3 028 486 472 Bytes	3 028 550 448 Bytes	+62.5 KiB	+0.002%
ydbd stripped size	480 129 624 Bytes	480 149 080 Bytes	+19.0 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

[github-actions[bot]]

:white_circle: 2024-10-16 15:10:49 UTC Pre-commit check linux-x86_64-release-asan for 0c83a9a26cafd16f60c9fa9e734bc155f96aec9e has started. :white_circle: 2024-10-16 15:11:01 UTC Artifacts will be uploaded here :white_circle: 2024-10-16 15:14:46 UTC ya make is running... :yellow_circle: 2024-10-16 16:54:42 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
16737	16487	0	68	79	103

:green_circle: 2024-10-16 16:56:08 UTC Build successful. :yellow_circle: 2024-10-16 16:56:41 UTC ydbd size 5.7 GiB changed* by +1.6 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: f1972dba95b6239b6daec4c82203200d7a3c062d	merge: 0c83a9a26cafd16f60c9fa9e734bc155f96aec9e	diff	diff %
ydbd size	6 128 234 192 Bytes	6 129 873 336 Bytes	+1.6 MiB	+0.027%
ydbd stripped size	1 529 060 304 Bytes	1 529 416 688 Bytes	+348.0 KiB	+0.023%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

[github-actions[bot]]

:white_circle: 2024-10-17 09:19:51 UTC Pre-commit check linux-x86_64-release-asan for eb1b083c66bc7864227bff2366f56c0cd7e60d09 has started. :white_circle: 2024-10-17 09:20:01 UTC Artifacts will be uploaded here :white_circle: 2024-10-17 09:23:48 UTC ya make is running... :yellow_circle: 2024-10-17 11:23:23 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
16741	16357	0	84	239	61

:green_circle: 2024-10-17 11:24:48 UTC Build successful. :yellow_circle: 2024-10-17 11:25:24 UTC ydbd size 5.7 GiB changed* by +182.1 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: d5d5380056d1cefa66db888755a58226293e7d83	merge: eb1b083c66bc7864227bff2366f56c0cd7e60d09	diff	diff %
ydbd size	6 129 992 248 Bytes	6 130 178 712 Bytes	+182.1 KiB	+0.003%
ydbd stripped size	1 529 429 392 Bytes	1 529 487 408 Bytes	+56.7 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

[github-actions[bot]]

:white_circle: 2024-10-17 09:20:43 UTC Pre-commit check linux-x86_64-relwithdebinfo for eb1b083c66bc7864227bff2366f56c0cd7e60d09 has started. :white_circle: 2024-10-17 09:20:52 UTC Artifacts will be uploaded here :white_circle: 2024-10-17 09:24:40 UTC ya make is running... :green_circle: 2024-10-17 10:53:08 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
79631	65666	0	0	13870	95

:green_circle: 2024-10-17 11:00:41 UTC Build successful. :green_circle: 2024-10-17 11:01:00 UTC ydbd size 2.8 GiB changed* by +62.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: d5d5380056d1cefa66db888755a58226293e7d83	merge: eb1b083c66bc7864227bff2366f56c0cd7e60d09	diff	diff %
ydbd size	3 028 533 168 Bytes	3 028 596 792 Bytes	+62.1 KiB	+0.002%
ydbd stripped size	480 133 080 Bytes	480 152 792 Bytes	+19.2 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

[github-actions[bot]]

:white_circle: 2024-10-18 15:41:48 UTC Pre-commit check linux-x86_64-relwithdebinfo for 008320071999ddaf02120aba5362760620451cad has started. :white_circle: 2024-10-18 15:41:59 UTC Artifacts will be uploaded here :red_circle: 2024-10-18 15:43:53 UTC Graph compare failed, see the logs.

[github-actions[bot]]

:white_circle: 2024-10-18 15:41:58 UTC Pre-commit check linux-x86_64-release-asan for 008320071999ddaf02120aba5362760620451cad has started. :white_circle: 2024-10-18 15:42:08 UTC Artifacts will be uploaded here :red_circle: 2024-10-18 15:44:00 UTC Graph compare failed, see the logs.

[github-actions[bot]]

:white_circle: 2024-10-18 15:53:26 UTC Pre-commit check linux-x86_64-release-asan for 72c9ddb6f9c8082d07debfd4c77540cc02276b0b has started. :white_circle: 2024-10-18 15:53:36 UTC Artifacts will be uploaded here :white_circle: 2024-10-18 15:57:22 UTC ya make is running... :black_circle: 2024-10-18 15:57:31 UTC Check cancelled

[github-actions[bot]]

:white_circle: 2024-10-18 15:53:34 UTC Pre-commit check linux-x86_64-relwithdebinfo for 72c9ddb6f9c8082d07debfd4c77540cc02276b0b has started. :white_circle: 2024-10-18 15:53:45 UTC Artifacts will be uploaded here :black_circle: 2024-10-18 15:57:30 UTC Check cancelled

[github-actions[bot]]

:white_circle: 2024-10-18 16:00:32 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8f9ee26642d1dfd589acfdf0fcd5555ceb4540c1 has started. :white_circle: 2024-10-18 16:00:43 UTC Artifacts will be uploaded here :white_circle: 2024-10-18 16:04:31 UTC ya make is running... :red_circle: 2024-10-18 16:07:44 UTC Build failed, see the logs. Also see fail summary

[github-actions[bot]]

:white_circle: 2024-10-18 16:01:42 UTC Pre-commit check linux-x86_64-release-asan for 8f9ee26642d1dfd589acfdf0fcd5555ceb4540c1 has started. :white_circle: 2024-10-18 16:01:52 UTC Artifacts will be uploaded here :white_circle: 2024-10-18 16:05:40 UTC ya make is running... :red_circle: 2024-10-18 16:10:56 UTC Build failed, see the logs. Also see fail summary

[github-actions[bot]]

:white_circle: 2024-10-18 16:23:16 UTC Pre-commit check linux-x86_64-relwithdebinfo for f6b6c8f488c166ccd8be481e4f9caff09838dac5 has started. :white_circle: 2024-10-18 16:23:27 UTC Artifacts will be uploaded here :white_circle: 2024-10-18 16:27:18 UTC ya make is running... :red_circle: 2024-10-18 17:18:20 UTC Build failed, see the logs. Also see fail summary

[github-actions[bot]]

:white_circle: 2024-10-18 16:23:17 UTC Pre-commit check linux-x86_64-release-asan for f6b6c8f488c166ccd8be481e4f9caff09838dac5 has started. :white_circle: 2024-10-18 16:23:27 UTC Artifacts will be uploaded here :white_circle: 2024-10-18 16:27:15 UTC ya make is running... :red_circle: 2024-10-18 17:19:33 UTC Build failed, see the logs. Also see fail summary

[MBkkt]

Я потыкал из любопытства немного, из интересного мне ответили, и сказали что в принципе можно заюзать dom builder чтобы сразу в binary json строить (проблема что патчить нужно) https://github.com/simdjson/simdjson/discussions/2279#discussioncomment-10988814

Еще я немного привел к виду который в его примерах, и поправил всякие мелочи, вроде стало побыстрее, но депендс от данных наверно https://github.com/swalrus1/ydb/compare/experiment-simdjson-binary...MBkkt:ydb:ondemand-bench?expand=1

[github-actions[bot]]

:white_circle: 2024-10-19 20:42:24 UTC Pre-commit check linux-x86_64-relwithdebinfo for 597d629d06097136c0b5919d63b8421512812908 has started. :white_circle: 2024-10-19 20:42:36 UTC Artifacts will be uploaded here :white_circle: 2024-10-19 20:46:39 UTC ya make is running... :black_circle: 2024-10-19 20:47:49 UTC Check cancelled

[github-actions[bot]]

:white_circle: 2024-10-19 20:42:39 UTC Pre-commit check linux-x86_64-release-asan for 597d629d06097136c0b5919d63b8421512812908 has started. :white_circle: 2024-10-19 20:42:49 UTC Artifacts will be uploaded here :white_circle: 2024-10-19 20:46:40 UTC ya make is running... :black_circle: 2024-10-19 20:47:49 UTC Check cancelled