Toil submits all jobs twice, initially to do work then to (clean up?) -- second submission should only request 1 core

[jsmedmar]

Hi all,

I've noticed that my Toil (3.19.0) pipelines submit all jobs twice (as also noted by other people https://github.com/DataBiosphere/toil/issues/2730#issuecomment-509018961, @ionox0).

In the first submission, the job does the work. In the second, it seems it only cleans up the DAG (or something like this, e.g. resolve dependencies, etc...).

I think the second submission should only request 1 core.

This is because if the cluster is really busy, lets say submitting a 32 cores job might wait a long time in the queue due to resource availability. However, is known ahead of time that this job won't be doing any work.

What do you guys think? Could you point me to where in the code the second submission happens?

Thank you so much in advanced

┆Issue is synchronized with this Jira Story ┆friendlyId: TOIL-594

[DailyDreaming]

@jsmedmar Apologies for the slow reply. That issue also seems to have been running 3.19.0, though I don't know of any PR that specifically fixed the issue, there have been some major refactors since 3.19.0 came out. Have you tested this with the latest version of toil (preferably source)?

[mr-c]

Fixed in #4259

[unito-bot]

➤ Adam Novak commented:

Looks like I reopened this, because I thought the bot was mistakenly closing it. But it turns out that this was fixed already and the bot was closing it because it finally noticed it was closed on Gitlab.