Conditionally populate resourceLabels to tag field in AWS Batch job definition

[rbartelme-pivot]

New feature

Proposed change, if resourceLabels are present for a given process. They should populate the AwsBatchTaskHandler's tag's field when Nextflow registers a job definition on AWS Batch.

Usage scenario

Multiple users have asked for this feature to track resource utilization and this can be used by AWS Solutions Architects to manage Nextflow behavior in non-NF Tower systems. This also allows AWS Solutions Architects to track costs, compute utilization, and optimize deployments across AWS Batch's resources like Fargate and EC2.

Suggest implementation

Add a conditional to the nf-amazon plugin's groovy source code where if resourceLabels are present for a given process that is trying to register an AWS Batch template, the resourceLabels contents populate the tags via req.addTagsEntry().

https://github.com/nextflow-io/nextflow/blob/711864fd4fcdbf6839f0370fb609d28a003b6942/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchTaskHandler.groovy#L657

[pditommaso]

It sounds reasonable

[pditommaso]

Discussing a bit about this, this has not been done because the job definition is created by nextflow only the very first, and then re-used for all following runs using the same container.

Adding the labels, it should be recreated ever time a label value change, that may be expected, it could result in a proliferation of batch job definitions, which may not be desirable in common cases

[rbartelme-pivot]

Discussing a bit about this, this has not been done because the job definition is created by nextflow only the very first, and then re-used for all following runs using the same container.

Adding the labels, it should be recreated ever time a label value change, that may be expected, it could result in a proliferation of batch job definitions, which may not be desirable in common cases

Fair assessment of common use cases.

Would the tags not be able to simply be updated if the AWS IAM role allows for it? Does Nextflow create new template registrations with new Nextflow version releases or does it update the registrations?

[dapiam]

I agree that it sounds reasonable in almost all scenarios, but AWS Batch allows tags to be propagated when the job definition is created. Right now, this flag is being disabled.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-batch-jobdefinition.html

Perhaps it would be interesting to add an option to enable or disable this feature?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.