Migration tasks cannot restart when platform nodes crash

[iccaprar]

We have encountered the following situation when running commerce-db-sync on CCv2:

• a migration task was in progress
• platform node crashed and was restarted by the CCv2 orchestrator
• on platform restart, CronJobManager#getRunningOrRestartedCronJobsForNode sets the migration cronjob to ABORTED
• we tried to restart the cronjob from Backoffice (but a trigger ends up in the same situation)
• the cronjob starts but never does anything, just waits forever for a sync task to complete

On further investigation, we found out that the incremental job implementation is checking if there is an existing running migration and if yes, it just starts waiting for that to finish. But that running migration is not actually running, as the thread running it was long gone with the node restart.

When we run some DB checks, this is the data we find:

SELECT * FROM MIGRATIONTOOLKIT_TABLECOPYSTATUS

migrationId	startAt	endAt	lastUpdate	total	completed	failed	status
b3be66a1-e52d-44c0-bc73-f84ed4e5f4e7	2023-11-14 12:00:48.0833333		2023-11-14 12:00:50.616	1	0	0	RUNNING

SELECT * FROM MIGRATIONTOOLKIT_TABLECOPYTASKS

targetnodeId	migrationId	pipelinename	sourcetablename	targettablename	columnmap	duration	sourcerowcount	targetrowcount	failure	error	published	truncated	lastupdate	avgwriterrowthroughput	avgreaderrowthroughput	copymethod	keycolumns	durationinseconds
13	b3be66a1-e52d-44c0-bc73-f84ed4e5f4e7	sapbwentry->SAPBWENTRY	sapbwentry	SAPBWENTRY	{}		9183	0	0		0	0	2023-11-14 12:00:50.616	0.00	0.00			0.00

We have to go and manually change the status in these tables to error:

UPDATE MIGRATIONTOOLKIT_TABLECOPYTASKS SET failure=1 WHERE migrationid='b3be66a1-e52d-44c0-bc73-f84ed4e5f4e7';

UPDATE MIGRATIONTOOLKIT_TABLECOPYSTATUS set status='ABORTED' where migrationid='b3be66a1-e52d-44c0-bc73-f84ed4e5f4e7';

Then when we start the cronjob, it does not find a running task, it creates a new on and works correctly.

[iccaprar]

We implemented a local fix for this, which we can submit to the repo:

• we listen for AfterCronJobCrashAbortEvent that is sent by the CronJobManager
• when we get this event, we look for running tasks and mark them as aborted like above

Since we have a single DB migration possible, it works for us.

An even better approach would be to add on the MigrationCronJob a new attribute with the migrationId, so when we get the event that the cronjob was aborted, we can cancel only the needed migration task.

[lnowakowski]

We've recently fixed some newly discovered issues regarding resuming of failed migration (#14), main issue was due to logic fetching pending failed copy tasks, with condition of cluster node ID, which was resulting invalid or even empty results, especially after node restart (if there are no fixed cluster ID assigned).

If you already have something implemented to handle cronjob restart case, please either create a PR here, or point me to your fork (if possible) where you added such change.

Adding migration ID to cronjob model data might be obviously quite useful in multiple cases, starting with proper abort/failure handling. We could also attach migration report log to cronjob execution log, or at least reference from job, to report download location via Backoffice UI component or something similar.