spidermon icon indicating copy to clipboard operation
spidermon copied to clipboard

Add handling for datetime aware start time in PeriodicExecutionTimeMonitor

Open further-reading opened this issue 2 years ago • 0 comments

In the current version of scrapy, the code below breaks due to spider start time being timezone aware https://github.com/scrapinghub/spidermon/blob/master/spidermon/contrib/scrapy/monitors/monitors.py#L497

now = datetime.datetime.utcnow()
start_time = self.data.stats.get("start_time")
duration = now - start_time

It currently throws the following error:

TypeError: can't subtract offset-native and offset-aware datetimes

Backwards compatible Fix: add try/except to add utc timezone to now timestamp when error thrown.

further-reading avatar Oct 15 '23 13:10 further-reading