Add Azure specifc Sarek pipeline config

[vsmalladi]

Currently this is part of the Azure specific config for Sarek, but can be optimized.

azure {
    batch {
        pools {
            auto {
               autoScale = true

               sku = "batch.node.centos 7"

               offer = "centos-container"

               publisher = "microsoft-azure-batch"

               vmType = 'Standard_E64_v3'

               vmCount = 1

            }
        }
    }
}

[adamrtalbot]

What would the benefit of having a Sarek specific Azure config be over updating the general Azure Batch config?

[maxulysse]

Agreeing with @adamrtalbot not sure why this has to be sarek specific.

[vsmalladi]

@maxulysse @adamrtalbot the pipeline specific config will have the vm_types that are a recomendation to scale for cost and time on azure for specific pipelies as the default settings don't translate well for azure currently.

[adamrtalbot]

Given Azure VM sizes are region, account and time specific, it would make more sense to make it a parameter you can set. We don't have any E64s in our quota and I do not have the budget to increase that!

params {
    azure_vm_type = "Standard_E64_v3"
}

azure {
    batch {
        pools {
            auto {
                vmType = params.azure_vm_type
            }
        }
    }
}

[vsmalladi]

I agree. I think a parameter that can set different vm_types with a default if available.

Thanks, Venkat

[adamrtalbot]

I have something that is already pretty close, if I open a PR could you give it some testing for me?

[vsmalladi]

@adamrtalbot yes I can do that. Are you thinking pipeline specific or generic to all pipelines?

[adamrtalbot]

@adamrtalbot yes I can do that. Are you thinking pipeline specific or generic to all pipelines?

Take a look here: https://github.com/nf-core/configs/pull/406

[adamrtalbot]

Woops, I pushed to this repo instead of my fork. Didn't realise I had permission to do that. Hopefully that's OK.

[maxulysse]

Woops, I pushed to this repo instead of my fork. Didn't realise I had permission to do that. Hopefully that's OK.

I think we have protection on the main branch, so you can push on a different one. But could be worth checking this out @nf-core/core

[ewels]

Yeah, master should be protected..

[maxulysse]

@adamrtalbot do you think we can close this one?

[adamrtalbot]

Yes. I believe @vsmalladi has an even better version?

[vsmalladi]

Yes will add one soon