Memory of process exceed limit. Start execute plan fragment

[xyan3880]

We are testing the loading performance using http and CSV format (about 100MB, 50k rows per request). After a certain loads, the BE crashed with "Memory of process exceed limit" error, like this:

"Memory limit exceeded: Memory of process exceed limit. read and decompress page Used: 28153092936, Limit: 27300971151. Mem usage has exceed the limit of BE"

We restarted BE but it refuses to process any further loading requests, even a small load after a few hours. All requests are timed out and eventually return the "Memory of process exceed limit" error.

Please see the attached BE logs. be.WARNING.txt

Any suggestion to "avoid" overloading BE? And how can we recover from such state? We also want the option to limit the total memory BE can use, because the system will have some other processes running.

Thanks.

Steps to reproduce the behavior (Required)

repeat loading CSV data using http requests to the BE directly. Primary key table with auto-increment ID.

Expected behavior (Required)

1. if BE can not handle the load, the request should be rejected or failed.
2. BE should not crash
3. after crash, BE should be able to recover and process new requests.
4. BE should not use 28G ram while loading 100MB data pieces.

Real behavior (Required)

BE crashed and no longer in service even after restart.

StarRocks version (Required)

3.1.4-0c4b2a3

[milletnis]

It seems we are facing the same or a very similar issue. After some hours of (massive) parallel stream load we are reaching the BE memory limits. Also our BE processes stop working because they reached memory limits.

It also seems the limit which is parameterized is not working? (please check picture) Please also note the very high number of loads which consume memory?? We should have less than 100 parallel stream loads)

It seems that at a specific point in time the stream loading stops and memory is accumulating for one of the BEs. Please see graph for loaded rows and load_mem_bytes

Our setup: 3FE, each 1 core and 24 GB memory 3BE, each 16 core and 128 GB memory

Note: We are using PK-models for all tables -> but PK/UPDATE related monitoring metrics look fine to us

Starrocks version: 3.2.1-79ee91d

Our non-default params for BEs are: (not sure if all of these really make sense, especially "publish-version_woker_count") base_compaction_num_threads_per_disk = 4 cumulative_compaction_num_threads_per_disk = 8 cumulative_compaction_check_interval_seconds = 2 number_tablet_writer_threads = 32 flush_thread_num_per_store = 16 transaction_publish_version_worker_count = 32

[ayush-san]

Were you able to resolve this? I had a 5 BE cluster, and after 30 days, I started getting the same issue. I added two more be nodes, thinking it was because of the high concurrent query and data load, but I still receive the same issue.

[xyan3880]

Unfortunately we didn't resolve the issue and this stays a protentional deal-breaker.

We are going to try

• increase the ram of the nodes (not an option for hardware based systems)
• delete some of the latest data versions (tablets) and deal with the data loss. This requires digging into the source code and identify the data to be deleted.
• restrict the loading rate based on hardware information. (this is really bad for real-time system)

of course, we are still waiting for suggestions from StarRocks team.

[ayush-san]

I updated the following properties

transaction_apply_worker_count=8 // default max cores
update_compaction_size_threshold=134217728 // default 256M
disable_column_pool = true
disable_storage_page_cache = true

[Hatlassian]

I found that storage page cache uses up a significant amount of memory and doesn't really get purged. If you need to keep memory usage low it's definitely worth disabling it, or setting a lower limit for it using BE configuration storage_page_cache_limit which is by default 20%.

You can configure the total BE's memory limit using configuration mem_limit.

https://docs.starrocks.io/docs/administration/management/BE_configuration/#mem_limit

https://docs.starrocks.io/docs/administration/management/resource_management/Memory_management/#memory-classification

[miguelgilmartinez]

Are you using "enable_persistent_index" = "false" on table definition? Ingesting into PRIMARY KEY tables with that attribute set to false eats all memory up because indexes are kept on memory instead of disk. Check this: https://github.com/StarRocks/starrocks/pull/15729

[xyan3880]

No, the table is a primary-key table with about 120 columns, primary key on bigint id and timestamp, with these properties:

PROPERTIES (
"replication_num" = "3",
"in_memory" = "false",
"enable_persistent_index" = "true",
"replicated_storage" = "true",
"compression" = "LZ4"
); 

Under regular load, the BE memory usage is reasonable even with larger data set. We saw this problem when we put the database under heavy load, possibly a lot of frequent writes, the BE crashed even with a smaller database size. What really concerns us is that BE failed to restart even we stopped the load and restarted the database, even the whole system. Restarting the BE is definitely our highest priority since a total data loss is unacceptable.

[miguelgilmartinez]

Our use case: It was impossible for us to ingest from parquet to PRIMARY KEY table. But doing to DUPLICATE and then moving to PRIMARY with an INSERT INTO SELECT FROM was right. It took several hours for a 600GB table and the memory was ok along the full process.

[Hatlassian]

Thanks @miguelgilmartinez , I tried your strategy and was able to improve our writes to primary key table by going through a duplicate key table first. I was able to go from ~35k records per second directly into the primary key table, to ~150k records per second into the duplicate key table, then ~200k records per second from duplicate key to primary key table merge, even with a merge condition.

Sample merge condition:

INSERT INTO primary_key_table
SELECT updates.*
FROM updates
LEFT JOIN primary_key_table USING (primary_key_column)
WHERE
  updates.version >= primary_key_table.verison
  OR primary_key_table.version IS NULL;