hyperqueue (>=0.17.0) and Nextflow interpret memory resource differently

[bguo068]

Bug report

Expected behavior and actual behavior

Expected behavior: hyperqueue and nextflow should have same interpretation of memory resources.

Actual behavior: When set memory 2.GB in Nextflow, the .command.run a work directory uses this:

#HQ --resource mem=2147483648

This is problematic as newer versions of hyperqueue interpret 2147483648 as 2147483648 Mb (2 PB) instead of 2147483648 Bytes(2GB). This results in the hyperqueue job waiting for the a big memory resource that would never be possible for most HPCs.

Steps to reproduce the problem

main.nf:

process A {
        executor 'hq'
        cpus 1
        memory 2.GB
        input: val(x)
        output: path("*.txt")
        script:
        """
        echo ${x} > ${x}.txt
        """
}

workflow{
        channel.fromList([1,2]) | A
}

Run the pipeline:

nextflow main.nf

Program output

Nextflow output:

 N E X T F L O W   ~  version 24.04.2

Launching `main.nf` [scruffy_cantor] DSL2 - revision: 091b7ae540

WARN: The support for HyperQueue is an experimental feature and it may change in a future release
executor >  hq (2)
[8e/ff2537] A (2) | 0 of 2                                  <--------------- Never finish

And it will stuck forever.

Example .command.run file:

#!/bin/bash
#HQ --name nf-A_(2)
# ....
#HQ --resource mem=2147483648               <------------------
#HQ --cpus 1
....

Environment

• Nextflow version: 24.04.2 build 5914
• Java version: 17.0.7 2023-04-18 LTS
• Operating system: Linux
• Bash version: GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
• hyperqueue: v0.19.0 <-----------------------

Additional context

break change in hyperqueue about mem resource