sge cluster requirements

[jordandekraker]

I'm getting this issue on hippunfold_v2 but its likely true in previous version too, i haven't tested it.

i'm trying to run on a sge cluster, but had some trouble setting up my profile. I kept getting an error

hippunfold: error: unrecognized arguments: all

even when this was not in my sge profile or the hippunfold default profile. Eventually I tracked it down to https://github.com/khanlab/hippunfold/blob/5011dfa92d97b348fd7da6e2f7967891cf692ca2/hippunfold/config/snakebids.yml#L11, and when i replace this with '' it works.

2) A second issue along the way

is that i needed to

pip install snakemake-executor-plugin-cluster-generic

and set up a ~/.config/snakemake/sge/config.yaml file like this:

executor: cluster-generic
cluster-generic-submit-cmd: "qsub -terse -V -cwd -N smk.{rule}.{jobid} \
  -pe smp {threads} -l h_rt={resources.time_h}:00:00 \
  -l mem_free={resources.mem_mb}M -q mica.q \
  -o logs/{rule}.{jobid}.out -e logs/{rule}.{jobid}.err"
jobs: 64
latency-wait: 120
use-conda: true
conda-frontend: mamba
default-resources:
  - mem_mb=8000
  - time_h=6

note here that -q mica.q is specific to my current lab, but the rest should be fairly generic. Now instead of --cores X, I can run with --profile sge

I think we should include a "cluster computing" page in the Docs to overview this for SGE and SLURM. To make things easier, we could likely add snakemake-executor-plugin-cluster-generic as a dependency (or even as a snakebids dependency?), but i'm still not sure what to do about participant: all in the config/snakebids.yml

[akhanf]

For your first issue, it could be related to how the CLI is being formed, as snakebids will just use that as the target rule. You could try using verbose and keeping the lots to see what CLI is being run and where 'all' is being placed.. Snakemake chooses the first rule is '' is supplied which is 'all', but then presumably things will still be broken for you for other participant-levels? Maybe that doesn't matter..

For second, I haven't had a need to use the generic cluster executor since we've just been using the slurm executor on clusters ( which doesn't require an extra profile).. Could make sense to have some cluster executors as a dependency in hippunfold especially if we provide some specific instructions or example profiles for them. The plugins are better as a dependency in the downstream apps (like hippunfold), rather than in snakebids itself though..