FluidFramework
FluidFramework copied to clipboard
Prevent fluid-build workers from exhausting all of memory
Description
This makes three changes to how fluid-build handles memory use:
- When setting an infinite limit (unlimited) memory, encode that as infinity bytes, instead of -1 bytes to simply logic and make semantics more clear.
- Use os.freemem to lower worker memory use limit as a heuristic
- If there is less than 2 GB free, don't reuse the worker.
Also more documentation is added, including help text for the previously undocumented --worker flag.
These changes are sufficient to make the default configuration of fluid-build with --worker not run my system out of memory.
My system has 64 GB or ram, and was running out of memory when using --worker to clean build client before this change. With this change my system uses almost all the memory, and just swaps a tiny bit and runs about as fast as with a manually tuned -workerMemoryLimitMB 2000 (162 seconds). This is faster than without worker (~200 seconds) which uses far less memory.
Generally systems with more logical cpu cores (I have 32) get more memory consumption with fluid-build due to having more threads and thus more workers. This change should prevent such high core count systems from running out of memory in most cases when using --worker if they have enough memory to do a normal parallel build.
Reviewer Guidance
The review process is outlined on this wiki page.