resource usage!

[ctb]

on a Jetstream m1.medium, second run of nema-test (so, all software installed etc. etc.)

/usr/bin/time -v reported:

        Command being timed: "./run_eelpond nema-test full"
        User time (seconds): 1637.94
        System time (seconds): 706.61
        Percent of CPU this job got: 93%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 41:38.69
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 3926492
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 545
        Minor (reclaiming a frame) page faults: 288161380
        Voluntary context switches: 958956
        Involuntary context switches: 161188
        Swaps: 0
        File system inputs: 179680
        File system outputs: 22846136
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

41 minutes, 4 GB of RAM, 120 MB of disk space used, plus 14 GB of databases (presumably these are the dammit databases?)

Looking into the software, this is a nearly clean miniconda install followed by the eelpond run, so:

5.6 GB in /opt/miniconda/pkgs - this is probably the downloaded pkgs 1.3 GB in /opt/minoconda/envs/eelpond/ - this is the installed software.

So I think we can probably recommend that you have 25 GB of free space after conda install in order to even begin to run eelpond (b/c of databases), + space for reads and assembly and so on.

[bluegenes]

Can you check the .snakemake folder within the eelpond dir? I'm not in a conda env on my local machine, so that has 8.4 GB (presumably the installed pkgs for envs). I assume this would instead be part of /opt/miniconda/pkgs if in a conda env to start.

[ctb]

ooooh yes. already deleted it, will look again.

[ctb]

on jetstream m1.medium from an image w/eelpond packages pre-downloaded (home directory wiped, rest fine):

        Command being timed: "./run_eelpond nema-test full"
        User time (seconds): 3039.17
        System time (seconds): 868.13
        Percent of CPU this job got: 95%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 1:07:51
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 6581560
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 2606
        Minor (reclaiming a frame) page faults: 294534126
        Voluntary context switches: 1111573
        Involuntary context switches: 175526
        Swaps: 0
        File system inputs: 1662720
        File system outputs: 52137784
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0```

Odd, six gigs of RAM and 1 hr 10 minutes. Huh.

Well, anyway, that's what I got.

And for @bluegenes -

% du -sh .snakemake 4.8G .snakemake

[luizirber]

If you want finer details (for each step) you can also use the benchmark feature: https://snakemake.readthedocs.io/en/stable/tutorial/additional_features.html#benchmarking