[Crash/Broken] Memory Issues after 5000 Inserts

[def-roth]

Bug Description:

Manticore crashes after approx. ~4000 Insertions without delay and ~ 7000 Insertions with a 200ms waiting time every 100 insertions. Restarting the docker container is not possible and wiping /var/lib/manticore the only option.

The data consists of of 3000 records

title:: string ~300 bytes body:: string ~6kb

Database Schema:

The database schema consists of two filled columns and 32 unfilled test columns.

INFO: The schema is for testing only and does not resemble the final use in any way.

title: string attribute title_vec: float_vector knn_type='hnsw' knn_dims='1' hnsw_similarity='COSINE' hnsw_m='1' body: string stored indexed body_vec: float_vector knn_type='hnsw' knn_dims='3' hnsw_similarity='COSINE' hnsw_m='2' u1: float u2: timestamp u3: timestamp u4: string attribute u5: string stored indexed u6: float_vector knn_type='hnsw' knn_dims='1' hnsw_similarity='IP' hnsw_m='1' u7: string attribute u8: string stored indexed u9: float_vector knn_type='hnsw' knn_dims='1' hnsw_similarity='L2" hnsw_m='1' u10: string attribute u11: string stored indexed u12: float_vector knn_type='hnsw' knn_dims='1' hnsw_similarity='L2" hnsw_m='1' u13: string stored indexed u14: string attribute indexed u15: string attribute u16: json u17: json u18: json u19: string attribute u20: float u21: boolean u22: string attribute indexed u23: float_vector u24: multi u25: multi u26: json u27: json u28: json u29: float_vector u30: multi

Insertion

Insertion is done via JSON. Average insertion time: 45ms

"localhost:9308/insert"
{
    index: table,
    id: 0,
    doc: {
      title: title_str, 
      body: body_str
    }
}

Installation

docker run -e EXTRA=1 --name manticore -v $(pwd)/data:/var/lib/manticore -p 127.0.0.1:9306:9306 -p 127.0.0.1:9308:9308 -d manticoresearch/manticore

Crash Error Message

[Wed Jul 10 08:15:11.038 2024] [1] using config file '/etc/manticoresearch/manticore.conf.sh' (351 chars)...
starting daemon version '6.3.2 c296dc7c8@24062606 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)' ...
listening on all interfaces for mysql, port=9306
listening on UNIX socket /var/run/mysqld/mysqld.sock
listening on all interfaces for sphinx and http(s), port=9308
listening on 172.17.0.2:9312 for sphinx and http(s)
prereading 0 tables
preread 0 tables in 0.000 sec
accepting connections
[BUDDY] started v2.3.10 '/usr/share/manticore/modules/manticore-buddy/bin/manticore-buddy --listen=http://0.0.0.0:9308 --bind=127.0.0.1  --threads=8 --skip=manticoresoftware/buddy-plugin-sharding --skip=manticoresoftware/buddy-plugin-queue' at http://127.0.0.1:40585
[BUDDY] Loaded plugins:
[BUDDY]   core: empty-string, backup, emulate-elastic, create, insert, alias, select, show, cli-table, plugin, test, alter-distributed-table, alter-rename-table, modify-table, knn, replace
[BUDDY]   local: 
[BUDDY]   extra: 
  1900K .......... .......... .......... .......... ........  100% 29.7M=0.09sterminate called after throwing an instance of 'std::runtime_error'
  what():  Not enough memory: addPoint failed to allocate linklist
Crash!!! Handling signal 6

Restart Error Message

Manticore 6.3.2 c296dc7c8@24062606 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)
[Wed Jul 10 07:57:08.110 2024] [1] using config file '/etc/manticoresearch/manticore.conf.sh' (351 chars)...
starting daemon version '6.3.2 c296dc7c8@24062606 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)' ...
listening on all interfaces for mysql, port=9306
listening on UNIX socket /var/run/mysqld/mysqld.sock
listening on all interfaces for sphinx and http(s), port=9308
listening on 172.17.0.2:9312 for sphinx and http(s)
Manticore 6.3.2 c296dc7c8@24062606 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)

precaching table 'ms_29'
binlog: replaying log /var/lib/manticore/binlog/binlog.001
binlog: table ms_29: recovered from tid 0 to tid 6959
binlog: replay stats: 6959 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 1 tables
binlog: finished replaying /var/lib/manticore/binlog/binlog.001; 46.9 MB in 0.863 sec
binlog: replaying log /var/lib/manticore/binlog/binlog.002
binlog: replay stats: 0 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 0 tables
binlog: finished replaying /var/lib/manticore/binlog/binlog.002; 0.0 MB in 0.000 sec
binlog: finished replaying total 2 in 0.863 sec
prereading 1 tables
preread 1 tables in 0.000 sec
accepting connections
terminate called after throwing an instance of 'std::runtime_error'
  what():  Not enough memory: addPoint failed to allocate linklist
Crash!!! Handling signal 6
[BUDDY] started v2.3.10 '/usr/share/manticore/modules/manticore-buddy/bin/manticore-buddy --listen=http://0.0.0.0:9308 --bind=127.0.0.1  --threads=8 --skip=manticoresoftware/buddy-plugin-sharding --skip=manticoresoftware/buddy-plugin-queue' at http://127.0.0.1:44021
[BUDDY] Loaded plugins:
[BUDDY]   core: empty-string, backup, emulate-elastic, create, insert, alias, select, show, cli-table, plugin, test, alter-distributed-table, alter-rename-table, modify-table, knn, replace
[BUDDY]   local: 
[BUDDY]   extra: 

Notice

Using a local version without docker leads to the same error but a full purge and reinstall is sometimes necessary.

Manticore Search Version:

'6.3.2 c296dc7c8

Operating System Version:

Ubuntu 22.04.4 LTS

Have you tried the latest development version?

None

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• [ ] Implementation completed
• [ ] Tests developed
• [ ] Documentation updated
• [ ] Documentation reviewed
• [ ] Changelog updated