drush icon indicating copy to clipboard operation
drush copied to clipboard

migrate:import --delete also rolls back skipped rows

Open chrisolof opened this issue 3 years ago • 0 comments

Describe the bug When migrate:import's --delete flag initiates a rollback, all skipped records in the ID Map are deleted as well. Example:

 [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) in 1.2 seconds (0/min) - done with 'foo'
 [notice] 2 items are missing from source and will be rolled back
 5827/5827 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%
 [notice] Rolled back 2721 items - done with 'foo'

Note how it says it will roll back 2 items, yet actually rolls back 2721.

That leaves the migration in an incomplete state with unprocessed items:

 ------------------ -------- ------- -------------- ------------- --------------- 
  Migration ID       Status   Total   Imported       Unprocessed   Last Imported  
 ------------------ -------- ------- -------------- ------------- --------------- 
  foo                Idle     5827    3108 (53.3%)   2719                         
 ------------------ -------- ------- -------------- ------------- ---------------

If I run this migration a second time, the skipped rows are added back into the ID map and I'm once again at 100% processed:

5827/5827 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%
 [notice] Processed 2719 items (0 created, 0 updated, 0 failed, 2719 ignored) in 23.6 seconds (6924.9/min) - done with 'foo'
------------------ -------- ------- -------------- ------------- --------------------- 
  Migration ID       Status   Total   Imported       Unprocessed   Last Imported        
 ------------------ -------- ------- -------------- ------------- --------------------- 
  foo                Idle     5827    3108 (53.3%)   0             2022-02-03 17:07:31  
 ------------------ -------- ------- -------------- ------------- ---------------------

Ideally only the records (and ID Map rows) now missing from the source would be deleted, leaving the unprocessed column at 0.

To Reproduce Set up a migration that will skip some source rows. Import all source rows with it - note how many rows were skipped. Remove one or two rows from the source. Re-run this migration with the --delete flag. See that both the removed rows, as well as the previously skipped rows are rolled back. See that the migration now has unprocessed rows (via migrate:status).

Expected behavior When migrate:import's --delete flag initiates a rollback, only records missing from the source are deleted. Previously-skipped, processed rows should be left alone unless they are actually missing from the source.

Actual behavior Records missing from the source and any skipped, previously-processed, rows are rolled back.

Workaround Run the migration twice. First run will leave it in an incomplete state with unprocessed rows. The second run will re-process the skipped rows, skipping them again but bringing them back into the map table. The migration will then have zero unprocessed rows and allow dependent migrations to run.

System Configuration

Q A
Drush version? 11.0.4
Drupal version? 9.3.3
PHP version 7.4
OS? Linux

Additional information To get the delete flag working in 11.x I had to first apply the change in #5063.

chrisolof avatar Feb 04 '22 00:02 chrisolof