OpenWPM icon indicating copy to clipboard operation
OpenWPM copied to clipboard

Reimplement non relaxed shutdown

Open vringar opened this issue 5 years ago • 2 comments

With the storage rewrite we lost the ability to quickly save out data in case of an imminent shutdown. We should restore this capability as our shutdowns now usually take longer than 30s which would mean our container getting SIGKILLed in case it needs to shutdown in an inorderly manner.

vringar avatar Jan 25 '21 10:01 vringar

When exploring the issue in previous crawls we didn't observe any container being shut down in such a manner.

vringar avatar Jan 25 '21 10:01 vringar

To save time during shutdown the following steps should be followed:

  • No save_record task should be awaited
  • All currently not finalized visit_ids should be marked as unfinished
  • The cache should be flushed
  • The completion queue needs to filled appropriately

vringar avatar Feb 01 '21 10:02 vringar