Updating the modules.json file to new format results in branch lookup error

[mahesh-panchal]

Description of the bug

On this branch https://github.com/mahesh-panchal/Earth-Biogenome-Project-pilot/tree/update_prepare_input when I use the command nf-core modules update -a in Gitpod: https://gitpod.io/#https://github.com/mahesh-panchal/Earth-Biogenome-Project-pilot/tree/update_prepare_input, fails to update the modules.json file to the new format. It prompts me for the following:

INFO     The 'modules.json' file is not up to date. Recreating the 'module.json' file.                                                                                      modules_json.py:439
INFO     Was unable to find matching module files in the master branch.                                                                                                     modules_json.py:231
? Was the modules installed from a different branch in the remote? (Use arrow keys)
 » No
   bump-kbtools
   Test_unsetJAVA
   humann
   elixir
   greater-than-256
   merge
   fastqvalidator
   jwarnn-patch-1
   trimmomatic
   head

and then cannot find the branch "No".

Command used and terminal output

$ nf-core modules update -a

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.5.1 - https://nf-co.re


WARNING  modules.json entry {'blast/blastn': {'git_sha': 'e745e167c1020928ef20ea1397b6b4d230681b4d'}, 'busco': {'git_sha': '9fa6b6c1fc930830c1d819a9273b4ec12ffacb0a'},     modules_json.py:373
         'diamond/blastx': {'git_sha': 'bd3bfe0817246082525ab93707976676b1fe208b'}, 'fastk/fastk': {'git_sha': '4e308c131e29bc2cca3d46ba8f7e59e9d3785198'}, 'fastk/histex':                    
         {'git_sha': 'fd5aea0b76c1e5b908191f212382a55a57cee316'}, 'genescopefk': {'git_sha': 'e91e99db30c143ed913313301b53b5454df8720d'}, 'genomescope2': {'git_sha':                          
         'd54cf467f47461e8304e6fc4a4a1192869bfd792'}, 'mash/screen': {'git_sha': 'c452f562890534ca3d0a331de7c35b744037fdae'}, 'mash/sketch': {'git_sha':                                       
         'e745e167c1020928ef20ea1397b6b4d230681b4d'}, 'merquryfk/katcomp': {'git_sha': '280eec531785bdba28faf01bd6dd6b41fabccb8e'}, 'merquryfk/katgc': {'git_sha':                             
         '233fa70811a03a4cecb2ece483b5c8396e2cee1d'}, 'merquryfk/merquryfk': {'git_sha': '8ded6d441abe1a111c20482fa50ad3837b1747bf'}, 'merquryfk/ploidyplot': {'git_sha':                      
         '0af6adef555e54c2dcf0343646325864cacb5176'}, 'meryl/count': {'git_sha': 'a43bc940d6aaeaa755b34484cffeb5b41bce92eb'}, 'meryl/histogram': {'git_sha':                                   
         '129f57ccd9adc9016aaa3f2e2e6ab556ba8de7b6'}, 'meryl/unionsum': {'git_sha': '57fee2b3d6759c36f2a3e9181ab5f309cf62b6f1'}, 'minimap2/align': {'git_sha':                                 
         '1a5a9e7b4009dcf34e6867dd1a5a1d9a718b027b'}, 'quast': {'git_sha': 'd473a247d2e0c619b0df877ea19d9a5a98c8e3c8'}, 'samtools/fastq': {'git_sha':                                          
         '897c33d5da084b61109500ee44c01da2d3e4e773'}, 'seqkit/split2': {'git_sha': 'e745e167c1020928ef20ea1397b6b4d230681b4d'}} does not have a git_url or modules entry                       
INFO     The 'modules.json' file is not up to date. Recreating the 'module.json' file.                                                                                      modules_json.py:439
INFO     Was unable to find matching module files in the master branch.                                                                                                     modules_json.py:231
? Was the modules installed from a different branch in the remote? No
ERROR    Branch 'No' not found in 'nf-core/modules'                                                                                                                             __main__.py:540

System information

The environment is the Gitpod environment where:

• nextflow -version: version 22.04.0 build 5697
• Hardware: Cloud
• OS: Unix
• version of nf-core tools: 2.5.1
• Python 3.9.12

[mirpedrol]

Hi @mahesh-panchal! I fixed this questionary in #1831. In your case the module is not found because the files are different from the ones in the modules repo. The follow up question after selecting "No" is if you want to move the module to the "local" directory or remove it. I checked as examples the modules fastk/fastk and fask/merge and the differences are minimal, two comments and a new line. I think this would be a case where we can use the new patch command, so I would add a new option to maintain the files as they are and try to find the commit sha from the modules repo, but I was wondering if you think there's a different case that you would like to handle here. For more context, this questions happen during the automatic update of modules.json before updating the modules. Module update will happen right after this.

[nvnieuwk]

Hi, I got the same error on my pipeline. I updated the template and did a manual merge with a development branch. I got a lot of merge conflicts and solved these, but got another error if I tried to install a module:

$ nf-core modules install tabix/bgziptabix

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.5.1 - https://nf-co.re


ERROR    'base_path'

Apparently I forgot to add the branch to the modules in modules.json. (thanks to @maxulysse for spotting this issue).

So the old modules.json looked something like this:

{
    "name": "institute/pipeline",
    "homePage": "https://github.com/institute/pipeline",
    "repos": {
        "nf-core/modules": {
            "git_url": "https://github.com/nf-core/modules.git",
            "modules": {
                "bcftools/concat": {
                    "git_sha": "51b0a6e4f3b048a1eebeb86896fa0374771ca554"
                },
                "bcftools/convert": {
                    "git_sha": "8d4373b4e8a5d41fbcfd131388e9fea687858683"
                }
            }
        }
    }
}

While it should look like this:

{
    "name": "institute/pipeline",
    "homePage": "https://github.com/institute/pipeline",
    "repos": {
        "nf-core/modules": {
            "git_url": "https://github.com/nf-core/modules.git",
            "modules": {
                "bcftools/concat": {
                    "branch": "master",
                    "git_sha": "51b0a6e4f3b048a1eebeb86896fa0374771ca554"
                },
                "bcftools/convert": {
                    "branch": "master",
                    "git_sha": "8d4373b4e8a5d41fbcfd131388e9fea687858683"
                }
            }
        }
    }
}

[mahesh-panchal]

Hi @mahesh-panchal! I fixed this questionary in #1831. In your case the module is not found because the files are different from the ones in the modules repo. The follow up question after selecting "No" is if you want to move the module to the "local" directory or remove it. I checked as examples the modules fastk/fastk and fask/merge and the differences are minimal, two comments and a new line. I think this would be a case where we can use the new patch command, so I would add a new option to maintain the files as they are and try to find the commit sha from the modules repo, but I was wondering if you think there's a different case that you would like to handle here. For more context, this questions happen during the automatic update of modules.json before updating the modules. Module update will happen right after this.

I didn't get a follow up question when I did this. It just stopped at the error. You can try this in the Gitpod session. The modules are different because they're older versions I think. There's no need to keep them, and no other scenario to consider here.