configs Custom config for nf-core eager on Crick's data and cluster

name: New Config about: A new cluster config

Please follow these steps before submitting your PR:

[x] If your PR is a work in progress, include [WIP] in its title
[x ] Your PR targets the master branch
[ ] You've included links to relevant issues, if any

Steps for adding a new config profile:

[x] Add your custom config file to the conf/ directory
[ ] Add your documentation file to the docs/ directory
[ ] Add your custom profile to the nfcore_custom.config file in the top-level directory
[ ] Add your custom profile to the README.md file in the top-level directory
[ ] Add your profile name to the profile: scope in .github/workflows/main.yml

Mar 15 '22 14:03 alexandregilardet

Hi @alexandregilardet I'm a bit confused what you're trying to do here.

What you want to do is in the configs/conf/pipeline/eager directory, make a new file called crick.config, and paste in the contents of your local custom_eager.config file (with a header like in eva or maestro etc.)

Then add the entry to this file under configs/pipeline/eager.config.

And finally mention in the docs/crick.md mention that there is a specific crick-specific profile for eager.

Mar 16 '22 07:03 jfy133

Header and filename are corrected.

Mar 16 '22 15:03 alexandregilardet

The test doesn't work: Cannot find revision 2.4.2-- Make sure that it exists in the remote repositoryhttps://github.com/nf-core/eager``

Also, I'm a bit confused because there's already a general crick profile for the cluster in general (not eager specific) which I also generally use as -profile screening,crick, how would it work now? -profile crick,crick but you've got to select the screening/production/external as well?

Mar 18 '22 16:03 alexandregilardet

The test doesn't work: Cannot find revision 2.4.2-- Make sure that it exists in the remote repositoryhttps://github.com/nf-core/eager``

Also, I'm a bit confused because there's already a general crick profile for the cluster in general (not eager specific) which I also generally use as -profile screening,crick, how would it work now? -profile crick,crick but you've got to select the screening/production/external as well?

Oops, sorry - you might need to Nextflow pull once, or just use whatever version you're using normally.

Basically if you go -profile crick a Nextflow pipeline will load the general one. It'll then look for any pipeline specific ones - if it finds one, it'll overwrite anything also specified in the general one. So in this regard you don't have to do anything.

Then yes, for the specific ones you just need to spend too the end:

-profile crick,screening (in that order)

Mar 18 '22 16:03 jfy133

Alexandre is not with the Crick anymore I think, what shall we do with this PR @jfy133 ? @alexandregilardet anyone we can ping about this one?

Oct 04 '22 20:10 apeltzer

Sorry I didn't get to finish that one before leaving, maybe @ChristopherBarrington or @pontussk (with my replacement) can help?

Oct 05 '22 08:10 alexandregilardet

any update on that @ChristopherBarrington @pontussk ?

Mar 02 '23 11:03 maxulysse

hi @maxulysse I am not entirely sure what this PR is about. It looks like it is to add default resource requests to eager when processing our data on our cluster, which at the moment are placed in a separate config file.

I am looking after running the pipeline as an interim before Alex's replacement starts; I would be hesitant to make changes to the workflow that I have been given.

I can see there are 2 failing tasks but to be honest I don't know what I need to do to help here. If it is a case of running something on our cluster I can do that (there may be some daft questions along the way though!). Otherwise, would it be ok for me to tell the new bioinformatician about this when they start so they can make all the changes they need?

Mar 03 '23 08:03 ChristopherBarrington

hi @maxulysse I am not entirely sure what this PR is about. It looks like it is to add default resource requests to eager when processing our data on our cluster, which at the moment are placed in a separate config file.

I am looking after running the pipeline as an interim before Alex's replacement starts; I would be hesitant to make changes to the workflow that I have been given.

I can see there are 2 failing tasks but to be honest I don't know what I need to do to help here. If it is a case of running something on our cluster I can do that (there may be some daft questions along the way though!). Otherwise, would it be ok for me to tell the new bioinformatician about this when they start so they can make all the changes they need?

No worries about the failed task, the branch was just out of date.

Mar 03 '23 09:03 maxulysse

If it is a case of running something on our cluster I can do that (there may be some daft questions along the way though!). Otherwise, would it be ok for me to tell the new bioinformatician about this when they start so they can make all the changes they need?

@ChristopherBarrington If you can just run nextflow run nf-core/eager -profile test,crick --custom_config_base "https://raw.githubusercontent.com/nf-core/configs/custom-eager-crick" and see if it runs through, that would be enough for us to merge, and then the replacement can run it on 'real' data to check it is optimised at a later :)

Mar 03 '23 12:03 jfy133

Sorry for the delay - I can't run the test because the profiles are not found in that directory.

[babs001 eager] > nextflow run nf-core/eager -profile test,crick --custom_config_base "https://raw.githubusercontent.com/nf-core/configs/custom-eager-crick"
N E X T F L O W  ~  version 22.10.3
Pulling nf-core/eager ...
 downloaded from https://github.com/nf-core/eager.git
WARNING: Could not load nf-core/config/eager profiles: https://raw.githubusercontent.com/nf-core/configs/custom-eager-crick/pipeline/eager.config
No such file: Config file does not exist: https://raw.githubusercontent.com/nf-core/configs/custom-eager-crick/pipeline/conf/test.config

Mar 08 '23 17:03 ChristopherBarrington

Unfortunatley this probably means there is an issue with the config itself (missing comma, or bracket etc), but unforatuntely NXF error handling is really bad...

you would have to do some local testing by deleting different blocks to see where the error is happening (I would try myself but odn't have capacity at the moment : ( )

Mar 09 '23 11:03 jfy133

OK. A quick look and there is an errant ' here:

https://github.com/nf-core/configs/blob/a3e0654c49900282fd461c05d81784cff852b7f3/conf/pipeline/eager/crick.config#L2

should I make a fork, edit, PR and then try the local command again? Or could you make that small edit and I will retest? :)

otherwise I can make a fork and investigate more closely asap

Mar 09 '23 11:03 ChristopherBarrington

OK. A quick look and there is an errant ' here:

https://github.com/nf-core/configs/blob/a3e0654c49900282fd461c05d81784cff852b7f3/conf/pipeline/eager/crick.config#L2

should I make a fork, edit, PR and then try the local command again? Or could you make that small edit and I will retest? :)

otherwise I can make a fork and investigate more closely asap

Done it fo ryou, but you can also make suggestions directly on this PR and we can push these :)

Mar 09 '23 12:03 jfy133

thanks for that. It looks like it is almost there. The requested change here is the name - so I need to put myself in as the contact.

config_profile_contact = "Christopher Barrington (@ChristopherBarrington)"

I tried to make that change but can't get around the "You’re making changes in a project you don’t have write access to. Submitting a change will write it to a new branch in your fork ChristopherBarrington/configs-1, so you can send a pull request.".

I made a comment on that line as well ... does that work?

Sorry for my lack of understanding here!

Mar 10 '23 12:03 ChristopherBarrington

Ah! You're probably not in the nf-core organisation ;)

I've sent an invite - you can accept and try again and it should work :)

Mar 10 '23 15:03 jfy133

The test profile completes ok:

> nextflow run nf-core/eager -r 2.4.2 -profile crick,test --custom_config_base 'https://github.com/nf-core/configs/raw/custom-eager-crick/'
###
-[nf-core/eager] Pipeline completed successfully-
-[nf-core/eager] MultiQC run report can be found in ./results/multiqc -
WARN: Graphviz is required to render the execution DAG in the given format -- See http://www.graphviz.org for more info.
Completed at: 13-Mar-2023 07:43:33
Duration    : 8m 13s
CPU hours   : (a few seconds)
Succeeded   : 26

Mar 13 '23 07:03 ChristopherBarrington

AWesome! Let me do one more review and we can merge in - thanks for the tsting @ChristopherBarrington !

Mar 13 '23 07:03 jfy133

Great, thanks. Is there some sort of prize for longest time to merge the smallest change?? Thank you for your patience with me though!

Mar 13 '23 07:03 ChristopherBarrington

Great, thanks. Is there some sort of prize for longest time to merge the smallest change?? Thank you for your patience with me though!

How about an e-cookie :cookie: ;). If you have permissions feel free to merge once you've decided a change based on my last comment or not.

Mar 13 '23 08:03 jfy133

@ChristopherBarrington also if you're not sure what ot add to the docs, we can just merge it in anyway ;)

Mar 13 '23 10:03 jfy133

I'll just merge this in now so it's available, can be updated in the future :)

Mar 13 '23 13:03 jfy133