pypiper icon indicating copy to clipboard operation
pypiper copied to clipboard

unify memory usage logging

Open stolarczyk opened this issue 6 years ago • 2 comments

The memory usage values for commands are more exact than the final memory report, which in a situation presented below is misleading. We should either make the command memory less precise or the final one more precise. Perhaps we should record bytes and just convert the number to a convenient form before logging, (e.g, MB). This function may be useful: https://github.com/pepkit/ubiquerg/blob/5926bb1da2baa5bccb93480368ec5bef9f7de8c2/ubiquerg/files.py#L58-L71

</pre>
Command completed. Elapsed time: 0:00:00. Running peak memory: 0.003GB.
  PID: 99358;	Command: cut;	Return code: 0;	Memory used: 0.002GB

> `touch /sfs/qumulo/qproject/shefflab/genomes_new/hg19/fasta/build_complete.flag` (99360)

<pre>
</pre>
Command completed. Elapsed time: 0:00:00. Running peak memory: 0.003GB.
  PID: 99360;	Command: touch;	Return code: 0;	Memory used: 0.0GB
Finished building asset 'fasta'

Changed status from running to completed.
Starting cleanup: 0 files; 0 conditional files for cleanup

> `Time`	0:14:10	refgenie	_RES_

> `Success`	08-07-09:09:32	refgenie	_RES_

##### [Epilogue:]
*   Total elapsed time:  0:01:41.620000
*     Peak memory used:  0.0 GB
* Pipeline completed at:  (08-07 09:09:32) elapsed: 850.0 _TIME_

stolarczyk avatar Aug 07 '19 13:08 stolarczyk

and the time certainly does not need to be this accurate

stolarczyk avatar Aug 07 '19 20:08 stolarczyk

agreed. I've been revamping the way the epilog is printed. fixed the time thing.

on memory, one other thing I noticed: we should report the peak memory for this run, and the peak memory for all runs. Given that we're already parsing the profile to do this for the time, we might as well get the cross-run peak memory as well.

nsheff avatar Aug 12 '19 16:08 nsheff