spidermon
spidermon copied to clipboard
Add handling for datetime aware start time in PeriodicExecutionTimeMonitor
In the current version of scrapy, the code below breaks due to spider start time being timezone aware https://github.com/scrapinghub/spidermon/blob/master/spidermon/contrib/scrapy/monitors/monitors.py#L497
now = datetime.datetime.utcnow()
start_time = self.data.stats.get("start_time")
duration = now - start_time
It currently throws the following error:
TypeError: can't subtract offset-native and offset-aware datetimes
Backwards compatible Fix: add try/except to add utc timezone to now timestamp when error thrown.