Django-link-archive icon indicating copy to clipboard operation
Django-link-archive copied to clipboard

Crawlee script support uses file system

Open rumca-js opened this issue 6 months ago • 0 comments

Crawlee uses file system to store some data. This might lead to errors if multiple processes use storage / crawlee os.environ["CRAWLEE_STORAGE_DIR"] = "./storage/{}".format(os.getpid())

I changed it so new instance of script uses a different dir, but there should be some point of time when this is cleared.

Should be cleared:

  • when django starts
  • ?

rumca-js avatar Aug 22 '24 14:08 rumca-js