ydb Create vector index in SchemeShard

Changelog entry

Support for vector index in SchemeShard.

Base table:

CREATE TABLE vectors (
    id Uint32,                      // id of a vector
    embedding String,               // vector embedding
    PRIMARY KEY (id)
)

Vector index table:

CREATE TABLE vector_index (
    level Uint8,		    // level of tree
    id Uint32,                      // id of the nearest vector of the cluster           
    embedding String,               // coordinate of the cluster
    ids String,                     // list of vector ids
    PRIMARY KEY (level, id)
)

Two columns are migrated from base table: id (key column) and embedding (data column). Two special columns are added: level (key column) and ids (data column).

Current limitations:

The base table primary key constitutes a single attribute and should be either Uint32 or Uint64. In this example it is id.

Changelog category

Not for changelog (changelog entry is not required)

Additional information

...

May 29 '24 10:05 azevaykin

:white_circle: 2024-05-29 10:52:15 UTC Pre-commit check for 73ba2aff6cc5ffe3b4b2a78dc8b221dda8c827cb has started. :white_circle: 2024-05-29 10:55:06 UTC Build linux-x86_64-release-asan is running... :green_circle: 2024-05-29 11:38:30 UTC Build successful. :white_circle: 2024-05-29 11:38:42 UTC Tests are running... :red_circle: 2024-05-29 13:37:55 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10858	10786	0	32	29	11

May 29 '24 10:05 github-actions[bot]

:white_circle: 2024-05-29 10:52:35 UTC Pre-commit check for 73ba2aff6cc5ffe3b4b2a78dc8b221dda8c827cb has started. :white_circle: 2024-05-29 10:55:32 UTC Build linux-x86_64-relwithdebinfo is running... :green_circle: 2024-05-29 11:38:10 UTC Build successful. :white_circle: 2024-05-29 11:44:07 UTC Tests are running... :red_circle: 2024-05-29 13:23:39 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
38400	33612	0	1	4782	5

May 29 '24 10:05 github-actions[bot]

:white_circle: 2024-05-29 10:52:40 UTC Pre-commit check for 73ba2aff6cc5ffe3b4b2a78dc8b221dda8c827cb has started. :white_circle: 2024-05-29 10:55:42 UTC Build linux-x86_64-release-clang14 is running... :green_circle: 2024-05-29 11:35:17 UTC Build successful.

May 29 '24 10:05 github-actions[bot]

:white_circle: 2024-05-30 06:24:45 UTC Pre-commit check for d3cd61857ff61f714892630e3c752143fd383b93 has started. :white_circle: 2024-05-30 06:27:37 UTC Build linux-x86_64-release-clang14 is running... :green_circle: 2024-05-30 07:06:09 UTC Build successful.

May 30 '24 06:05 github-actions[bot]

:white_circle: 2024-05-30 06:24:47 UTC Pre-commit check for d3cd61857ff61f714892630e3c752143fd383b93 has started. :white_circle: 2024-05-30 06:27:36 UTC Build linux-x86_64-release-asan is running... :green_circle: 2024-05-30 07:07:55 UTC Build successful. :white_circle: 2024-05-30 07:08:09 UTC Tests are running... :red_circle: 2024-05-30 08:47:21 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10774	10687	0	41	34	12

May 30 '24 06:05 github-actions[bot]

:white_circle: 2024-05-30 06:24:58 UTC Pre-commit check for d3cd61857ff61f714892630e3c752143fd383b93 has started. :white_circle: 2024-05-30 06:27:48 UTC Build linux-x86_64-relwithdebinfo is running... :green_circle: 2024-05-30 07:06:28 UTC Build successful. :white_circle: 2024-05-30 07:06:42 UTC Tests are running... :red_circle: 2024-05-30 08:34:56 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
38356	33551	0	12	4787	6

May 30 '24 06:05 github-actions[bot]

Комментарий от ijon:

нет поддержки нового типа индекса в public api и с++ sdk — но это пока ожидаемо, полагаю, ты этого пока и не делал (там помимо прочего должна быть конверсия в table_description.cpp, и, например, расширение enum в ytable_enum.h)
расширение EPathSubType как способ указать, что таблица не обычная пользовательская, а реализует что-то специфическое
- выдача подтипа в результате describe
- и сокрытие такой спец.таблицы от глаз пользователя на уровне scheme-cache — индексные таблицы у нас от пользователя прячутся

Jun 03 '24 09:06 azevaykin

:white_circle: 2024-06-13 05:36:53 UTC Pre-commit check for 113cc6cb2eb1609c9c0db1b40db951853b2be7cb has started. :white_circle: 2024-06-13 05:39:54 UTC Build linux-x86_64-release-clang14 is running... :black_circle: 2024-06-13 06:06:18 UTC Check cancelled

Jun 13 '24 05:06 github-actions[bot]

:white_circle: 2024-06-13 05:37:06 UTC Pre-commit check for 113cc6cb2eb1609c9c0db1b40db951853b2be7cb has started. :white_circle: 2024-06-13 05:40:09 UTC Build linux-x86_64-relwithdebinfo is running... :black_circle: 2024-06-13 06:06:18 UTC Check cancelled

Jun 13 '24 05:06 github-actions[bot]

:white_circle: 2024-06-13 05:37:08 UTC Pre-commit check for 113cc6cb2eb1609c9c0db1b40db951853b2be7cb has started. :white_circle: 2024-06-13 05:40:11 UTC Build linux-x86_64-release-asan is running... :black_circle: 2024-06-13 06:06:17 UTC Check cancelled

Jun 13 '24 05:06 github-actions[bot]

:white_circle: 2024-06-13 06:07:41 UTC Pre-commit check for 1800b4964e55f6b72ffc8338e553017a9fd3e3c8 has started. :white_circle: 2024-06-13 06:10:30 UTC Build linux-x86_64-release-clang14 is running... :green_circle: 2024-06-13 06:46:47 UTC Build successful.

Jun 13 '24 06:06 github-actions[bot]

:white_circle: 2024-06-13 06:07:42 UTC Pre-commit check for 1800b4964e55f6b72ffc8338e553017a9fd3e3c8 has started. :white_circle: 2024-06-13 06:10:36 UTC Build linux-x86_64-release-asan is running... :green_circle: 2024-06-13 06:52:14 UTC Build successful. :white_circle: 2024-06-13 06:52:31 UTC Tests are running... :red_circle: 2024-06-13 08:50:12 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15386	15284	0	41	48	13

Jun 13 '24 06:06 github-actions[bot]

:white_circle: 2024-06-13 06:10:15 UTC Pre-commit check for 1800b4964e55f6b72ffc8338e553017a9fd3e3c8 has started. :white_circle: 2024-06-13 06:13:10 UTC Build linux-x86_64-relwithdebinfo is running... :green_circle: 2024-06-13 06:53:29 UTC Build successful. :white_circle: 2024-06-13 06:53:43 UTC Tests are running... :red_circle: 2024-06-13 09:07:38 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
75042	62226	0	1	12804	11

Jun 13 '24 06:06 github-actions[bot]

:white_circle: 2024-06-20 13:28:35 UTC Pre-commit check for 3196274ff980fc5af9aab9216b13722871b040d4 has started. :white_circle: 2024-06-20 13:31:37 UTC Build linux-x86_64-release-asan is running... :green_circle: 2024-06-20 14:01:30 UTC Build successful. :white_circle: 2024-06-20 14:01:39 UTC Tests are running... :red_circle: 2024-06-20 15:56:25 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14791	14642	0	75	54	20

:yellow_circle: 2024-06-20 15:57:11 UTC ydbd size 5.7 GiB changed by +252.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: 053170694b7e6ba3b2c3b4bcde30dc61c1c02605	merge: 3196274ff980fc5af9aab9216b13722871b040d4	diff	diff %
ydbd size	6 149 731 584 Bytes	6 149 990 040 Bytes	+252.4 KiB	+0.004%
ydbd stripped size	1 527 571 776 Bytes	1 527 634 912 Bytes	+61.7 KiB	+0.004%

Jun 20 '24 13:06 github-actions[bot]

:white_circle: 2024-06-20 13:47:39 UTC Pre-commit check for 3196274ff980fc5af9aab9216b13722871b040d4 has started. :white_circle: 2024-06-20 13:50:30 UTC Build linux-x86_64-relwithdebinfo is running... :green_circle: 2024-06-20 14:21:14 UTC Build successful. :white_circle: 2024-06-20 14:21:24 UTC Tests are running... :red_circle: 2024-06-20 16:44:05 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
75345	61990	0	46	13291	18

:yellow_circle: 2024-06-20 16:44:47 UTC ydbd size 8.6 GiB changed by +298.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: 053170694b7e6ba3b2c3b4bcde30dc61c1c02605	merge: 3196274ff980fc5af9aab9216b13722871b040d4	diff	diff %
ydbd size	9 215 542 624 Bytes	9 215 847 784 Bytes	+298.0 KiB	+0.003%
ydbd stripped size	711 334 640 Bytes	711 366 416 Bytes	+31.0 KiB	+0.004%

Jun 20 '24 13:06 github-actions[bot]

:white_circle: 2024-06-20 13:48:04 UTC Pre-commit check for 3196274ff980fc5af9aab9216b13722871b040d4 has started. :white_circle: 2024-06-20 13:50:56 UTC Build linux-x86_64-release-clang14 is running... :green_circle: 2024-06-20 14:30:57 UTC Build successful.

Jun 20 '24 13:06 github-actions[bot]

:white_circle: 2024-06-27 07:57:41 UTC Pre-commit check for 10222bd6695a7be9360fa2ea8ee63002ade5674b has started. :white_circle: 2024-06-27 08:00:39 UTC Build linux-x86_64-release-clang14 is running... :red_circle: 2024-06-27 08:39:11 UTC Build failed. see the build logs.

Jun 27 '24 07:06 github-actions[bot]

:white_circle: 2024-06-27 07:57:45 UTC Pre-commit check for 10222bd6695a7be9360fa2ea8ee63002ade5674b has started. :white_circle: 2024-06-27 08:00:54 UTC Build linux-x86_64-release-asan is running... :red_circle: 2024-06-27 08:34:18 UTC Build failed. see the build logs.

Jun 27 '24 07:06 github-actions[bot]

:white_circle: 2024-06-27 07:57:55 UTC Pre-commit check for 10222bd6695a7be9360fa2ea8ee63002ade5674b has started. :white_circle: 2024-06-27 08:00:57 UTC Build linux-x86_64-relwithdebinfo is running... :red_circle: 2024-06-27 08:35:19 UTC Build failed. see the build logs.

Jun 27 '24 07:06 github-actions[bot]

:white_circle: 2024-07-01 10:33:24 UTC Pre-commit check for a01e7b4c2d541600afe3f537a03932a2446a5f55 has started. :white_circle: 2024-07-01 10:36:18 UTC Build linux-x86_64-release-clang14 is running... :red_circle: 2024-07-01 10:46:48 UTC Build failed. see the build logs.

Jul 01 '24 10:07 github-actions[bot]

:white_circle: 2024-07-01 11:07:08 UTC Pre-commit check for a01e7b4c2d541600afe3f537a03932a2446a5f55 has started. :white_circle: 2024-07-01 11:10:33 UTC Build linux-x86_64-release-asan is running... :red_circle: 2024-07-01 11:16:57 UTC Build failed. see the build logs. :red_circle: 2024-07-01 11:16:59 UTC Tests run skipped.

Jul 01 '24 11:07 github-actions[bot]

:white_circle: 2024-07-01 11:42:25 UTC Pre-commit check for a01e7b4c2d541600afe3f537a03932a2446a5f55 has started. :white_circle: 2024-07-01 11:45:59 UTC Build linux-x86_64-relwithdebinfo is running... :red_circle: 2024-07-01 11:53:51 UTC Build failed. see the build logs. :red_circle: 2024-07-01 11:53:52 UTC Tests run skipped.

Jul 01 '24 11:07 github-actions[bot]

:white_circle: 2024-07-02 14:21:34 UTC Pre-commit check for e90412730687e544d4f04c88ce3eb950b853220a has started. :white_circle: 2024-07-02 14:25:01 UTC Build linux-x86_64-relwithdebinfo is running... :green_circle: 2024-07-02 15:31:12 UTC Build successful. :white_circle: 2024-07-02 15:31:23 UTC Tests are running... :red_circle: 2024-07-02 17:54:52 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
76627	63230	0	2	13383	12

:green_circle: 2024-07-02 17:55:32 UTC ydbd size 8.3 GiB changed* by -229.6 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: 9e52af7523e338e8e984600110b46821fe303d0e	merge: e90412730687e544d4f04c88ce3eb950b853220a	diff	diff %
ydbd size	8 946 688 168 Bytes	8 946 453 024 Bytes	-229.6 KiB	-0.003%
ydbd stripped size	487 865 256 Bytes	487 904 520 Bytes	+38.3 KiB	+0.008%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

Jul 02 '24 14:07 github-actions[bot]

:white_circle: 2024-07-02 14:30:16 UTC Pre-commit check for e90412730687e544d4f04c88ce3eb950b853220a has started. :white_circle: 2024-07-02 14:33:58 UTC Build linux-x86_64-release-asan is running... :green_circle: 2024-07-02 15:27:10 UTC Build successful. :white_circle: 2024-07-02 15:27:25 UTC Tests are running... :red_circle: 2024-07-02 17:34:56 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15650	15037	0	96	344	173

:yellow_circle: 2024-07-02 17:35:34 UTC ydbd size 5.4 GiB changed* by +239.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: 29e3cb438ab5f6df73768785433c2506ee5263ec	merge: e90412730687e544d4f04c88ce3eb950b853220a	diff	diff %
ydbd size	5 829 838 496 Bytes	5 830 083 560 Bytes	+239.3 KiB	+0.004%
ydbd stripped size	1 251 538 728 Bytes	1 251 663 336 Bytes	+121.7 KiB	+0.010%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

Jul 02 '24 14:07 github-actions[bot]

:white_circle: 2024-07-02 14:33:13 UTC Pre-commit check for e90412730687e544d4f04c88ce3eb950b853220a has started. :white_circle: 2024-07-02 14:36:31 UTC Build linux-x86_64-release-clang14 is running... :green_circle: 2024-07-02 15:16:21 UTC Build successful.

Jul 02 '24 14:07 github-actions[bot]

:white_circle: 2024-07-08 07:39:27 UTC Pre-commit check for d95932f1525b86cae015148ff46f00642932b456 has started. :white_circle: 2024-07-08 07:42:35 UTC Build linux-x86_64-release-clang14 is running... :green_circle: 2024-07-08 08:20:21 UTC Build successful.

Jul 08 '24 07:07 github-actions[bot]

:white_circle: 2024-07-08 07:41:27 UTC Pre-commit check for d95932f1525b86cae015148ff46f00642932b456 has started. :white_circle: 2024-07-08 07:44:32 UTC Build linux-x86_64-release-asan is running... :green_circle: 2024-07-08 08:35:20 UTC Build successful. :white_circle: 2024-07-08 08:35:33 UTC Tests are running... :red_circle: 2024-07-08 10:47:08 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15680	15067	0	100	343	170

:yellow_circle: 2024-07-08 10:47:58 UTC ydbd size 5.2 GiB changed* by +313.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: 1d57390026f155f72eefcb959790ffe039195c98	merge: d95932f1525b86cae015148ff46f00642932b456	diff	diff %
ydbd size	5 614 136 416 Bytes	5 614 456 976 Bytes	+313.0 KiB	+0.006%
ydbd stripped size	1 209 988 632 Bytes	1 210 056 824 Bytes	+66.6 KiB	+0.006%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

Jul 08 '24 07:07 github-actions[bot]

:white_circle: 2024-07-08 07:41:42 UTC Pre-commit check for d95932f1525b86cae015148ff46f00642932b456 has started. :white_circle: 2024-07-08 07:44:42 UTC Build linux-x86_64-relwithdebinfo is running... :green_circle: 2024-07-08 08:53:53 UTC Build successful. :white_circle: 2024-07-08 08:54:09 UTC Tests are running... :green_circle: 2024-07-08 11:04:21 UTC Tests successful.

Test history | Test log

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
76743	63304	0	0	13428	11

:yellow_circle: 2024-07-08 11:05:13 UTC ydbd size 8.1 GiB changed* by +357.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: 1d57390026f155f72eefcb959790ffe039195c98	merge: d95932f1525b86cae015148ff46f00642932b456	diff	diff %
ydbd size	8 721 786 512 Bytes	8 722 152 480 Bytes	+357.4 KiB	+0.004%
ydbd stripped size	476 623 288 Bytes	476 645 688 Bytes	+21.9 KiB	+0.005%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

Jul 08 '24 07:07 github-actions[bot]

:white_circle: 2024-07-08 14:55:21 UTC Pre-commit check for 2e249ddd35fd66c8c7d64606037cdad2db999e9a has started. :white_circle: 2024-07-08 14:58:32 UTC Build linux-x86_64-release-clang14 is running... :green_circle: 2024-07-08 15:36:49 UTC Build successful.

Jul 08 '24 14:07 github-actions[bot]

:white_circle: 2024-07-08 14:55:24 UTC Pre-commit check for 2e249ddd35fd66c8c7d64606037cdad2db999e9a has started. :white_circle: 2024-07-08 14:58:44 UTC Build linux-x86_64-relwithdebinfo is running... :green_circle: 2024-07-08 16:01:35 UTC Build successful. :white_circle: 2024-07-08 16:01:54 UTC Tests are running... :red_circle: 2024-07-08 18:03:57 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
76745	63303	0	4	13428	10

:yellow_circle: 2024-07-08 18:04:33 UTC ydbd size 8.1 GiB changed* by +398.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: cc81c7eb0bdcd035ea66223a9d81bcc3c76f29ba	merge: 2e249ddd35fd66c8c7d64606037cdad2db999e9a	diff	diff %
ydbd size	8 709 596 768 Bytes	8 710 005 216 Bytes	+398.9 KiB	+0.005%
ydbd stripped size	475 600 376 Bytes	475 627 096 Bytes	+26.1 KiB	+0.006%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

Jul 08 '24 14:07 github-actions[bot]