amoro icon indicating copy to clipboard operation
amoro copied to clipboard

[IMPROVEMENT]AMS using too much memory

Open HuangFru opened this issue 3 years ago • 1 comments

When the AMS starts, it will use 400-500MB memory. When execute first sql in AMS Dashboard, MEM usage will rise to about 1GB. After writing some datas, then execute select * statement in AMS Dashboard.MEM usage will rise to about 1.7GB(increase about 700M). The second time execute select * statement in AMS Dashboard, MEM usage will rise to about 1.8GB(increase about 100M). The third time execute select * statement in AMS Dashboard, MEM usage will rise to about 1.85GB(increase about 50M or less). The forth time execute select * statement in AMS Dashboard, MEM usage will rise to about 1.90GB(increase about 50M or less).

Then first time execute insert overwrite in AMS Dashboard, MEM usage will increase about 300MB. Then second time execute insert overwrite in AMS Dashboard, MEM usage will increase about 50MB or less. Then third time execute insert overwrite in AMS Dashboard, MEM usage will increase about 50MB or less.

in conclusion: first time execute sql statement in AMS Dashboard, MEM usage will increase a lot(about 700MB), after first time it will be acceptable. first time execute different types DML sql statement in AMS Dashboard, MEM usage will increase about 100MB-300MB, after first time it will be acceptable.

Evaluate: just use quick start in docker, 3GB MEM is required for insurance. Flink containers will also use about 1.7GB MEM in docker quick start, 2GB MEM is required for insurance.

HuangFru avatar Aug 19 '22 03:08 HuangFru

  1. executing spark sql through ams dashboard is not a good choice, arctic shall connect kyuubi or trino in terminal module in future work
  2. there should be a memory limitation for ams local sql execution

majin1102 avatar Aug 19 '22 05:08 majin1102