Django-link-archive
Django-link-archive copied to clipboard
Crawlee script support uses file system
Crawlee uses file system to store some data. This might lead to errors if multiple processes use storage / crawlee os.environ["CRAWLEE_STORAGE_DIR"] = "./storage/{}".format(os.getpid())
I changed it so new instance of script uses a different dir, but there should be some point of time when this is cleared.
Should be cleared:
- when django starts
- ?