cubed icon indicating copy to clipboard operation
cubed copied to clipboard

Warn user when executing plan memory exceeds allowed memory.

Open alxmrs opened this issue 1 year ago • 1 comments

Fixes #110. I don't know if this is a good place to include this warning (during a call to compute()). Since array ops are lazily evaluated, there may not be a better place. Please let me know if you have ideas about where the warning could fit.

alxmrs avatar Nov 21 '23 04:11 alxmrs

Thanks for the PR @alxmrs!

The memory stuff is potentially confusing as there are quite a few different definitions in Cubed! So I'll try to explain what I was trying to achieve here. As it stands, Cubed will already warn the user if the projected memory exceeds allowed memory before running the computation (code, test).

What #110 is about is giving a warning if the actual memory usage exceeds allowed memory after the computation has run. This is referred to peak_measured_mem in https://tom-e-white.com/cubed/user-guide/memory.html.

The way it could be implemented would be to have a callback that collects all the peak_measured_mem_end values for all the tasks in a computation, then once the computation is over issues a warning if any exceed allowed_mem.

This is somewhat similar to the code in https://github.com/tomwhite/cubed/blob/d19caecdb148b46c89350123a6388adc39bff592/cubed/extensions/history.py, but the code there to analyze the memory usage is called manually by the user after the computation is over. The idea behind #110 is to warn the user without them having to configure anything.

tomwhite avatar Nov 21 '23 12:11 tomwhite

I've implemented this in #516

tomwhite avatar Jul 22 '24 21:07 tomwhite