pypiper
pypiper copied to clipboard
unify memory usage logging
The memory usage values for commands are more exact than the final memory report, which in a situation presented below is misleading. We should either make the command memory less precise or the final one more precise. Perhaps we should record bytes and just convert the number to a convenient form before logging, (e.g, MB). This function may be useful: https://github.com/pepkit/ubiquerg/blob/5926bb1da2baa5bccb93480368ec5bef9f7de8c2/ubiquerg/files.py#L58-L71
</pre>
Command completed. Elapsed time: 0:00:00. Running peak memory: 0.003GB.
PID: 99358; Command: cut; Return code: 0; Memory used: 0.002GB
> `touch /sfs/qumulo/qproject/shefflab/genomes_new/hg19/fasta/build_complete.flag` (99360)
<pre>
</pre>
Command completed. Elapsed time: 0:00:00. Running peak memory: 0.003GB.
PID: 99360; Command: touch; Return code: 0; Memory used: 0.0GB
Finished building asset 'fasta'
Changed status from running to completed.
Starting cleanup: 0 files; 0 conditional files for cleanup
> `Time` 0:14:10 refgenie _RES_
> `Success` 08-07-09:09:32 refgenie _RES_
##### [Epilogue:]
* Total elapsed time: 0:01:41.620000
* Peak memory used: 0.0 GB
* Pipeline completed at: (08-07 09:09:32) elapsed: 850.0 _TIME_
and the time certainly does not need to be this accurate
agreed. I've been revamping the way the epilog is printed. fixed the time thing.
on memory, one other thing I noticed: we should report the peak memory for this run, and the peak memory for all runs. Given that we're already parsing the profile to do this for the time, we might as well get the cross-run peak memory as well.