concurrency with deblur using `PLUGIN_LAUNCHER = qiita-plugin-launcher`

[sjanssen2]

I seem to constantly running into an concurrency issue. I've set up a local Qiita instance on a Ubuntu virtual machine, based on the example nginx configuration, i.e. with one master and three workers, and installed all plugins necessary for 16S processing.

Creating a test study with just 3 samples let me process up to the timming: However, deblur fails to finish. It executes the 4 deblur step with additional 4 SEPP steps but to my surprise, it seems to trigger two posts to /qiita_db/jobs/JOBID/complete/ from two different workers:

==> qiita_21177.log <==
[I 240301 15:55:36 web:2161] 200 POST /qiita_db/jobs/00e919d2-b9b6-458d-bc72-3bfb3cb3bb89/complete/ (127.0.0.1) 68.18ms

==> qiita_21175.log <==
[I 240301 15:55:40 web:2161] 200 POST /qiita_db/jobs/00e919d2-b9b6-458d-bc72-3bfb3cb3bb89/complete/ (127.0.0.1) 603.42ms

The admin error table lists Cannot change the step of a job whose status is not 'running', jobid=00e919d2-b9b6-458d-bc72-3bfb3cb3bb89

This might be due to the fact, that deblur generats two artifacts?! Referring to the different processes that are triggered https://github.com/qiita-spots/qiita/blob/13eafdf1a2eb0c8b5d4019ad294b7490cd396139/scripts/qiita-recover-jobs#L144-L155 I see that the jobid 00e919d2-b9b6-458d-bc72-3bfb3cb3bb89 belongs to the validate job of the reference-hit.biom artifact

SELECT * FROM qiita.processing_job WHERE processing_job_id = '00e919d2-b9b6-458d-bc72-3bfb3cb3bb89';:

          processing_job_id           |     email      | command_id |                                                                                                                                                                                                                                                                                                                                                                                                   command_parameters                                                                                                                                                                                                                                                                                                                                                                                                    | processing_job_status_id | logging_id |         heartbeat          |                            step                             | pending | hidden | external_job_id 
--------------------------------------+----------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------+------------+----------------------------+-------------------------------------------------------------+---------+--------+-----------------
 00e919d2-b9b6-458d-bc72-3bfb3cb3bb89 | [email protected] |         29 | {"analysis": null, "artifact_type": "BIOM", "files": "{\"biom\": [\"/home/qiita/Git/qiita-spots/qiita/qiita_db/support_files/production_data/working_dir/1ada410d-906a-4b44-b823-bfe97fa517fd/deblur_out/deblured/reference-hit.biom\"], \"preprocessed_fasta\": [\"/home/qiita/Git/qiita-spots/qiita/qiita_db/support_files/production_data/working_dir/1ada410d-906a-4b44-b823-bfe97fa517fd/deblur_out/deblured/reference-hit.seqs.fa\"], \"plain_text\": [\"/home/qiita/Git/qiita-spots/qiita/qiita_db/support_files/production_data/working_dir/1ada410d-906a-4b44-b823-bfe97fa517fd/deblur_out/deblured/insertion_tree.relabelled.tre\"]}", "name": "dflt_name", "provenance": "{\"job\": \"1ada410d-906a-4b44-b823-bfe97fa517fd\", \"cmd_out_id\": 13, \"name\": \"deblur reference hit table\"}", "template": 1} |                        6 |            | 2024-03-01 15:55:26.779568 | Completing via c806b5d8-ff9e-4fce-be70-781fcdbfe0ef [33894] |         | f      | 33490

• Where exactly are the POSTs to /qiita_db/jobs/JOBID/complete/ triggered?
• Why twice with the very same job.id?