qiita
qiita copied to clipboard
concurrency with deblur using `PLUGIN_LAUNCHER = qiita-plugin-launcher`
I seem to constantly running into an concurrency issue. I've set up a local Qiita instance on a Ubuntu virtual machine, based on the example nginx configuration, i.e. with one master and three workers, and installed all plugins necessary for 16S processing.
Creating a test study with just 3 samples let me process up to the timming:
However, deblur fails to finish. It executes the 4 deblur step with additional 4 SEPP steps but to my surprise, it seems to trigger two posts to
/qiita_db/jobs/JOBID/complete/
from two different workers:
==> qiita_21177.log <==
[I 240301 15:55:36 web:2161] 200 POST /qiita_db/jobs/00e919d2-b9b6-458d-bc72-3bfb3cb3bb89/complete/ (127.0.0.1) 68.18ms
==> qiita_21175.log <==
[I 240301 15:55:40 web:2161] 200 POST /qiita_db/jobs/00e919d2-b9b6-458d-bc72-3bfb3cb3bb89/complete/ (127.0.0.1) 603.42ms
The admin error table lists Cannot change the step of a job whose status is not 'running', jobid=00e919d2-b9b6-458d-bc72-3bfb3cb3bb89
This might be due to the fact, that deblur generats two artifacts?! Referring to the different processes that are triggered https://github.com/qiita-spots/qiita/blob/13eafdf1a2eb0c8b5d4019ad294b7490cd396139/scripts/qiita-recover-jobs#L144-L155 I see that the jobid 00e919d2-b9b6-458d-bc72-3bfb3cb3bb89
belongs to the validate
job of the reference-hit.biom artifact
SELECT * FROM qiita.processing_job WHERE processing_job_id = '00e919d2-b9b6-458d-bc72-3bfb3cb3bb89';
:
processing_job_id | email | command_id | command_parameters | processing_job_status_id | logging_id | heartbeat | step | pending | hidden | external_job_id
--------------------------------------+----------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------+------------+----------------------------+-------------------------------------------------------------+---------+--------+-----------------
00e919d2-b9b6-458d-bc72-3bfb3cb3bb89 | [email protected] | 29 | {"analysis": null, "artifact_type": "BIOM", "files": "{\"biom\": [\"/home/qiita/Git/qiita-spots/qiita/qiita_db/support_files/production_data/working_dir/1ada410d-906a-4b44-b823-bfe97fa517fd/deblur_out/deblured/reference-hit.biom\"], \"preprocessed_fasta\": [\"/home/qiita/Git/qiita-spots/qiita/qiita_db/support_files/production_data/working_dir/1ada410d-906a-4b44-b823-bfe97fa517fd/deblur_out/deblured/reference-hit.seqs.fa\"], \"plain_text\": [\"/home/qiita/Git/qiita-spots/qiita/qiita_db/support_files/production_data/working_dir/1ada410d-906a-4b44-b823-bfe97fa517fd/deblur_out/deblured/insertion_tree.relabelled.tre\"]}", "name": "dflt_name", "provenance": "{\"job\": \"1ada410d-906a-4b44-b823-bfe97fa517fd\", \"cmd_out_id\": 13, \"name\": \"deblur reference hit table\"}", "template": 1} | 6 | | 2024-03-01 15:55:26.779568 | Completing via c806b5d8-ff9e-4fce-be70-781fcdbfe0ef [33894] | | f | 33490
- Where exactly are the POSTs to
/qiita_db/jobs/JOBID/complete/
triggered? - Why twice with the very same
job.id
?