pycbc
pycbc copied to clipboard
Some PyCBC Offline jobs are still using too much memory
There was some work in/in advance of O4 to reduce the memory burden of plotting jobs when reading in the [HUGE] HDF_TRIGGER_MERGE files (often done via the coincident files).
However, there are still a small number of jobs that are using a large amount of memory (the following example probably is from a poorly tuned workflow, but you get the idea):
16103666 H 00:30 110 ┣━plot_singles_ID51_ID0017400 [plot_sing..017400.sh]
┗━MemoryUsage 60923 MB > RequestMemory 30000 MB.
16103668 H 00:43 110 ┣━plot_hist_ID58_ID0017406 [plot_hist..017406.sh]
┗━MemoryUsage 60923 MB > RequestMemory 8000 MB.
16103670 H 00:30 110 ┣━plot_singles_ID51_ID0017401 [plot_sing..017401.sh]
┗━MemoryUsage 60925 MB > RequestMemory 30000 MB.
16103671 H 00:31 110 ┣━plot_snrchi_ID49_ID0017388 [plot_snrc..017388.sh]
┗━MemoryUsage 85791 MB > RequestMemory 8000 MB.
16103638 H 00:39 10 ┃ ┣━merge_page_coi..o_ID0_PID1_ID1 [merge_pag..D1_ID1.sh: 5]
┗━MemoryUsage 16704 MB > RequestMemory 1980 MB.
So in particular plot_singles
, plot_hist
, plot_snrchi
and page_coi...
(whatever that one is!) should be looked at to see if they can have memory footprint reduced. Assigning to Gareth, but others might want to take this on!
Can this be closed?
Many of these are still using more memory than they should/could ... Just not enough that it's impossible to run them
OK, can you check if the list of codes involved should be shortened