Db store force triggered

[hjoliver]

These changes close #5020

Requirements check-list

• [x] I have read CONTRIBUTING.md and added my name as a Code Contributor.
• [x] Contains logically grouped changes (else tidy your branch by rebase).
• [x] Does not contain off-topic changes (use other PRs for other changes).
• [x] Applied any dependency changes to both setup.cfg and conda-environment.yml.
• [x] Appropriate tests are included (unit and/or functional).
• [x] Appropriate change log entry included.
• [x] No documentation update required.
• [ ] [if bugfix] PRs raised to both master and the relevant bugfix branch.

[MetRonnie]

Sorry, accidentally deleted 8.0.x!

[hjoliver]

Yes, I need to (re)-investigate that - I ~think I did it deliberately, and it seems to work correctly, but have forgotten my own chain of thought already... [DONE]

A good way to check updates of the task status table:

[scheduling]
    [[graph]]
        R1 = "a => b => c => d"
[runtime]
    [[a, b, d]]
        pre-script = sleep 5
    [[c]]   # trigger a new flow at a, then fail so the new flow will merge with me
        script = """
            sleep 5
            if ((CYLC_TASK_SUBMIT_NUMBER == 1 )); then
               cylc trigger --flow=new $CYLC_WORKFLOW_ID//1/a
               false
            fi
         """

Run the above with this alongside:

$ watch -n 1 sqlite3 -header -column ~/cylc-run/test/runN/log/db \
   "select name, flow_nums, submit_num, status from task_states"

[MetRonnie]

I tried your example running on 8.0.x and this branch, but I can't say I noticed a difference in the changes to the task_states table

[hjoliver]

Sorry, poorly explained by me! - that was just intended to show that my changes to the task status DB calls had not broken anything.

[hjoliver]

~~(TBD: resolve Oliver's question above.)~~ DONE

[MetRonnie]

Oops, this breaks the ability of Cylc to restart 8.0.{0,1,2} workflows.

ERROR - no such column: task_states.is_manual_submit
    Traceback (most recent call last):
      File "~/github/cylc-flow/cylc/flow/scheduler.py", line 670, in start
        await self.configure()
      File "~/github/cylc-flow/cylc/flow/scheduler.py", line 459, in configure
        self._load_pool_from_db()
      File "~/github/cylc-flow/cylc/flow/scheduler.py", line 719, in _load_pool_from_db
        self.workflow_db_mgr.pri_dao.select_task_pool_for_restart(
      File "~/github/cylc-flow/cylc/flow/rundb.py", line 860, in select_task_pool_for_restart
        for row_idx, row in enumerate(self.connect().execute(stmt)):
    sqlite3.OperationalError: no such column: task_states.is_manual_submit
CRITICAL - Workflow shutting down - no such column: task_states.is_manual_submit

We need to make this an 8.1.0-only bugfix.

• 8.0.x branch - #5173
• Master branch - #5174
• Both branches: add a functional test for restarting a min-compatible-version workflow?

Edit: Oliver has suggested an alternative is implementing a DB upgrader

---

Generally we need to be careful about changes to the DB structure.

[oliver-sanders]

Dammit, not sure how I missed that :facepalm:

[hjoliver]

Oops, sorry :face_with_spiral_eyes: should have thought of that myself!