improve container memory handling

[ixcat]

Recently discovered a compound set of issues when running the container on smaller systems (like the embedded VM / WSL used by Docker Desktop for windows) -

Basically, the container memory requirements can be higher than the base docker system in some cases, which will result in the server running properly until growth of the db memory results in the docker system running out of memory, at which point the Linux OOM killer kicks in, in many cases killing the DB process since it has a high likelihood of being the largest process in the host.

In the given example (https://github.com/arsenyf/FinkelsteinFontolan_2021NN/pull/8), tuning down the innodb_buffer_pool_size to 1GB still allows for the project queries to execute (e.g. there is sufficient mysql memory for query processing), but prevents the overall mysql process size from growing due to buffer pool caching and overwhelming the docker memory size, triggering the OOM killer.

While we can't tune for every situation, and having our expected defaults/minimum as a clear reference configuration variable is good, it would be good to improve the container reliability in these cases since it is often used as a component in user development and demonstration environments and the like. Some possible approaches:

1. Checking the memory available to the container and warning/exiting if it is insufficient (or configurably between warn/exit with one as default)
2. Everything in 1, plus startup scripting to dynamically adjust mysql config settings e.g. by warning and setting innodb_buffer_pool to 80% of memory, etc. The dynamic configuration itself and the tuning values could also be configurable for more flexibility.

and probably others

proposing:

• add logic to check actual memory against expected requirements and warn/exit (configurable with one being default)
• add auto-config logic with 2 knobs: 1 to enable or disable the auto-config, and another to control the value of innodb_buffer_pool_size, ideally as a percent and also a discrete G or M value.

side note: in many cases and in our usage buffer pool is responsible for the bulk of memory; in this test mysql container max was basically buffer_pool_size + ~300-500MB in the various tests (default 4G, 2G, 1G). This may vary by number of connections and other workload-specific factors but is probably a decent rule of thumb for the small-number-of-connections case typically involved when the issue occurs.

see also (windows specific):

https://docs.docker.com/docker-for-windows/#resources - Docker/Windows on WSL loses the memory setting; now done through WSL itself https://docs.microsoft.com/en-us/windows/wsl/wsl-config#configure-global-options-with-wslconfig : outlines default WSL memory space: 50% of total memory on Windows or 8GB, whichever is less; on builds before 20175: 80% of your total memory on Windows

[ixcat]

discussion/plan:

• confirm failure mode with compose / container limits (input for next-step decision)
• based on compose / container limit failure mode, proceed
• currently leaning towards retuning container for 'minimum spec', and adding logic to detect under-spec configurations