Ingestion of images stops unexpectedly after cleanup

[dhruvkb]

trafficstars

Description

All CI flakiness has boiled down to the ingestion of images stopping unexpectedly after the cleanup step. Rerunning ingestion almost surely fixes the issue.

We should investigate the issue and find out the cause. If it is incurable, we should add a step to retry the ingestion which is almost guaranteed to succeed the second time.

This also causes #562.

Reproduction

See failed runs:

• https://github.com/WordPress/openverse-api/runs/6527207725?check_suite_focus=true#step:8:702
• https://github.com/WordPress/openverse-api/runs/6527646327?check_suite_focus=true#step:11:480

Resolution

• [ ] 🙋 I would be interested in resolving this bug.

[dhruvkb]

#802 should reduce the flakiness in our CI by retrying ingestion if it unexpectedly stops but the real cause and the proper solution for this issue are yet to be found.

[dhruvkb]

^ Accidental click.

[zackkrida]

@WordPress/openverse-api have we observed this behavior in the API CI still?

[dhruvkb]

We covered over this problem by repeating the process if it fails (up to 3 retries) so this problem doesn't appear in CI anymore. But the underlying cause has not been fixed. @obulat's PR #839 removes the problematic step and that will fix the problem at the cause.