OpenWPM
OpenWPM copied to clipboard
Reimplement non relaxed shutdown
With the storage rewrite we lost the ability to quickly save out data in case of an imminent shutdown. We should restore this capability as our shutdowns now usually take longer than 30s which would mean our container getting SIGKILLed in case it needs to shutdown in an inorderly manner.
When exploring the issue in previous crawls we didn't observe any container being shut down in such a manner.
To save time during shutdown the following steps should be followed:
- No save_record task should be awaited
- All currently not finalized visit_ids should be marked as unfinished
- The cache should be flushed
- The completion queue needs to filled appropriately