django-scheduler
django-scheduler copied to clipboard
llazzaro
get_occurrences does not respect timezone across DST boundaries
It seems that there is no way to generate occurrences that are timezone-aware; for example, if a user creates an Event with a weekly recurrence rule and start = "2015-03-04T09:00:00-08:00" (9am PST), the generated occurrence for 2015-03-11 will be 1 hour off because dateutil.rrule does not support timezone in its icalendar implementation.
Has anybody come across this and found a solution that doesn't require forking dateutil or modifying the underlying ical implementation?
This is actually a limitation of pytz based on a implementation decision of dateutil (see PEP-431). It just does not work for tzinfo timezones with daylight time saving. Example, two datetimes, same zone only one in PST, the other in PDT, if converted to UTC they should have a 1 hour difference. The generated datetimes even land 7 minutes sooner than they should.
In [1]: import datetime
In [2]: from pytz import timezone
In [3]: utczone = timezone('UTC')
In [4]: pacific = timezone('US/Pacific')
In [5]: datetime_in_pst = datetime.datetime(2015,3,4,9,0,0, tzinfo=pacific)
In [6]: datetime_in_pdt = datetime.datetime(2015,3,11,9,0,0, tzinfo=pacific)
In [7]: datetime_in_pdt.astimezone(utczone)
Out[7]: datetime.datetime(2015, 3, 11, 16, 53, tzinfo=<UTC>)
In [8]: datetime_in_pst.astimezone(utczone)
Out[8]: datetime.datetime(2015, 3, 4, 16, 53, tzinfo=<UTC>)
pytz documentation mentions this limitation http://pytz.sourceforge.net/#localized-times-and-date-arithmetic.
The arrow library works around this issue as it does not rely on pytz for this and other reasons. It does still have other big issues with DST and I'm not sure if incorporating arrow here could solve this particular problem as there is a heavy dependency on dateutil in charge of generating the occurrences.
In [1]: import arrow
In [2]: datetime_in_pst = arrow.get(2015, 3, 4, 9, 0, 0, 0, 'US/Pacific')
In [3]: datetime_in_pdt = arrow.get(2015, 3, 11, 9, 0, 0, 0, 'US/Pacific')
In [4]: datetime_in_pst.to('UTC')
Out[4]: <Arrow [2015-03-04T17:00:00+00:00]>
In [5]: datetime_in_pdt.to('UTC')
Out[5]: <Arrow [2015-03-11T16:00:00+00:00]>
Second thought. There is a way to work with pytz that will make this work correctly, using localize as its being used in the code right now. I created a unit test for this and it actually passes. So this should not be an issue.
@aleontiev my guess is you`re creating the datetimes that are passed to Event (and feed the rrules) in a way its not supported by pytz, hence the generated datetimes in the rules do not work as expected.
Please notice the only way to correctly create a timezone-aware datetime is this:
>>> import pytz
>>> import datetime
>>>
>>> pacific = pytz.timezone('US/Pacific')
>>> pacific.localize(datetime.datetime(2015, 3, 11, 9, 0, 0))
datetime.datetime(2015, 3, 11, 9, 0, tzinfo=<DstTzInfo 'US/Pacific' PDT-1 day, 17:00:00 DST>)
>>> pacific.localize(datetime.datetime(2015, 3, 4, 9, 0, 0))
datetime.datetime(2015, 3, 4, 9, 0, tzinfo=<DstTzInfo 'US/Pacific' PST-1 day, 16:00:00 STD>)
Try to use this kind of datetimes in your code and let me know.
@mpaolino how can we apply this logic to Periods? Periods are useful for getting occurrences across multiple events, but if we were to use
Period(Event.objects.all, e_start, pacific.localize(datetime.datetime(2015, 3, 11, 10, 0)).get_occurrences()` (which is the Period equivalent of your event.get_occurrences test for this issue), the times returned are not localized.
EDIT: After checking out the Period class I've discovered the tzinfo flag, which addresses this problem. Should have talked to the rubber ducky first...
@shudson sorry for the lack of documentation. Let me know if you find any other problems with occurrences.
This still does not address the issue when you have a recurring event whose end recurring period ends after a DST switch.
I have all event times being stored as tz_aware but they all start creating wrong occurrences after a DST switch.
@nwaxiomatic can you please provide a concrete example with recurrence rule, start and end in a new issue so I can have a look at it? Please notice this issue is from 2015 (!) and probably should be closed (cc @llazzaro). Also be sure to use the latest version to date (0.8.3).
This code uses the UTC timezone as specified in the models file. Perhaps I am converting timezones incorrectly?
import pytz
import datetime
from django.utils import timezone
from schedule.models import Event, Occurrence, Calendar
from schedule.models.rules import Rule
utc = timezone.utc
pacific = pytz.timezone('US/Pacific')
start = utc.localize(datetime.datetime(2016, 10, 25, 9, 0, 0))
end = utc.localize(datetime.datetime(2016, 10, 25, 10, 0, 0))
pstart = pacific.localize(datetime.datetime(2016, 10, 25, 9, 0, 0))
pend = pacific.localize(datetime.datetime(2016, 10, 25, 10, 0, 0))
dstart = datetime.datetime(2016, 10, 25, 9, 0, 0)
dend = datetime.datetime(2016, 10, 25, 10, 0, 0)
rule = Rule(
name="test_rule",
description="daily",
frequency="DAILY"
)
calendar = Calendar(
name="test_calendar"
)
event = Event(
title="test_event",
calendar=calendar,
start=start,
end=end,
rule=rule
)
devent = Event(
title="test_event",
calendar=calendar,
start=pstart,
end=pend,
rule=rule
)
pevent = Event(
title="test_event",
calendar=calendar,
start=dstart,
end=dend,
rule=rule
)
o_start = utc.localize(datetime.datetime(2016, 11, 4, 9, 0, 0))
o_end = utc.localize(datetime.datetime(2016, 11, 7, 18, 0, 0))
occs = event.get_occurrences(o_start, o_end)
for occ in occs:
print occ.start.astimezone(pacific)
poccs = pevent.get_occurrences(o_start, o_end)
for occ in poccs:
print occ.start.astimezone(pacific)
doccs = devent.get_occurrences(o_start, o_end)
for occ in doccs:
print occ.start.astimezone(pacific)
and it prints out the following problematic info:
2016-11-04 02:00:00-07:00
2016-11-05 02:00:00-07:00
2016-11-06 01:00:00-08:00
2016-11-07 01:00:00-08:00
2016-11-04 02:00:00-07:00
2016-11-05 02:00:00-07:00
2016-11-06 01:00:00-08:00
2016-11-07 01:00:00-08:00
2016-11-04 09:00:00-07:00
2016-11-05 09:00:00-07:00
2016-11-06 08:00:00-08:00
2016-11-07 08:00:00-08:00
Please let me know if this is clear in terms of showcasing the issue and if there is any mistake in my conversion.
@nwaxiomatic hummm, yes, the problem is you're creating the occurrences in UTC timezone, and then converting them to Pacific, UTC is "constant" so occurrences will happily generate in constants intervals while Pacific time shifts. You need to generate occurrences using the correct timezone so dateutil generates them taking TZ shifts into consideration. In django-scheduler this translates to giving o_start and o_end a proper US/Pacific datetime instead of UTC (it probably deserves a proper method param). With your example, this prints:
2016-11-05 02:00:00-07:00
2016-11-06 02:00:00-08:00
2016-11-07 02:00:00-08:00
2016-11-04 09:00:00-07:00
2016-11-05 09:00:00-07:00
2016-11-06 09:00:00-08:00
2016-11-07 09:00:00-08:00
2016-11-04 09:00:00-07:00
2016-11-05 09:00:00-07:00
2016-11-06 09:00:00-08:00
2016-11-07 09:00:00-08:00
AFAIK these occurrences are correct. Beware how naive event dates are handled, they are set to the o_start timezone to generate the occurrences. So the last two events are in US/Pacific, and the first one had the event start set to 09:00 UTC which is 02:00 US/Pacific at that point in time. Also, the first occurrence for the first event will not show up in this example because event was set to start at 02:00 US/Pacific and I asked for occurrences after 09:00 US/Pacific.
IMHO this behaviour is correct but is probably better not to mix timezones when dealing with events and occurrences as it gets hairy very fast. Let me know if you think I missed something.
BTW, notice how 08:00:00-08:00 and 09:00:00-07:00 is exactly the same time in UTC.
Yes, I know. What I am saying is that standard code within the Event model defintion takes UTC to be the timezone if USE_TZ is set to True in the django settings. It should probably be set to use the actual timezone designated in the settings as well. Specifically this part:
def get_occurrence(self, date):
use_naive = timezone.is_naive(date)
tzinfo = timezone.utc
if timezone.is_naive(date):
date = timezone.make_aware(date, timezone.utc)
if date.tzinfo:
tzinfo = date.tzinfo
rule = self.get_rrule_object(tzinfo)
if rule:
next_occurrence = rule.after(tzinfo.normalize(date).replace(tzinfo=None), inc=True)
next_occurrence = tzinfo.localize(next_occurrence)
else:
next_occurrence = self.start
if next_occurrence == date:
try:
return Occurrence.objects.get(event=self, original_start=date)
except Occurrence.DoesNotExist:
if use_naive:
next_occurrence = timezone.make_naive(next_occurrence, tzinfo)
return self._create_occurrence(next_occurrence)
but there are many other cases where it makes this assumption
Or is it just that I need to make every form entry of dates to be timezone aware does django not do that on its own?
Sorry I know this could totally be my fault but it would be good to have some guidelines for people who want to use timezones in the future.
@nwaxiomatic don't know the rationale behind that :-/. In my case I generate all datetimes timezone aware, but that's me. I just fixed a couple of bugs in this project so it would work for my purposes, so I'm not even sure what are the implications of such a change. Maybe the maintainer has an idea @llazzaro.
@mpaolino I was under the assumption django created the datetimes fields as timezone aware if you had USE_TZ=True but maybe that is faulty reasoning.
Also a bit confused because in my example even when I use pstart (pacific localized time) the event still has the same issues
Yes you're right. It creates and stores them in UTC timezone aware dates (localtime converted to UTC). All my event dates are generated and stored in UTC, I just make sure to send the start and end datetimes with the correct timezone when I want my occurrences to generate.
OK, I now understand the problem, you must use get_occurrence with the proper TZ, not convert the time after you get the occurrence.
This is how I prepare new event dates in Event.save(). Django-scheduler and get_occurrence and get_occurrences do not adjust the date to DST changes, I'm afraid. I am not using Period, but the normal Event methods event_instance.get_occurrences() and event_instance.get_occurrence().
# assuming: self.start = '2019-10-18 14:00:00+02:00'
start_without_tz = parse(self.start, ignoretz=True)
self.start = get_localzone().localize(start_without_tz)
Did I overlook something?
I just noticed that saving it as I mentioned above is not encouraged in Django. Instead dates should be saved in UTC is in general and then means Django Schedule needs to adjust the UTC date to the appropriate timezone incl. DST. But...
Where and how does django-scheduler recognize the timezone it should output dates in? Let's assume I save a local date in DST (daylight savings time) as UTC in the DB and the DST ends, then django-scheduler will assume that occurrences which take place after the DST has ended, find place at the same UTC time, but in fact - cause they are local events - they take place one hour different than saved in the DB as UTC time. All generated occurrences after DST has ended still need to output the same local time.
How do I make this work in django-scheduler? Do I need to add a timezone-field to the event model and adjust the time of generated occurrences depending if the DST has changed or not?
Is there anyone who can help me out with this please? I need to make it work asap and am totally confused (and have been for quite a while) on how to make this work. Any chance @mpaolino or @llazzaro have some time to give me some hints please?
Just to reconfirm @mpaolino 's solution works, here is an example of weekly event starting 10AM every Sunday regardlessly. get_occurrence() with local time is consistent even daylight start.
import pytz
import datetime
from django.utils import timezone
from schedule.models import Event, Occurrence, Calendar
from schedule.models.rules import Rule
utc = timezone.utc
pacific = pytz.timezone('America/Los_Angeles')
start = utc.localize(datetime.datetime(2020, 1, 12, 18, 0, 0))
end = utc.localize(datetime.datetime(2020, 1, 12, 20, 0, 0)) #Janurary is PST
rule = Rule(
name="test_rule",
description="WEEKLY",
frequency="WEEKLY"
)
calendar = Calendar(
name="test_calendar"
)
event = Event(
title="test_event",
calendar=calendar,
start=start,
end=end,
rule=rule
)
o_start = pacific.localize(datetime.datetime(2021, 3, 1, 0, 0, 0))
o_end = pacific.localize(datetime.datetime(2021, 3, 31, 0, 0, 0)) # Daylight starts in Mar.14'21
occs = event.get_occurrences(o_start, o_end) # query with local time!!!
for occ in occs:
print(occ.start.astimezone(pacific))
####
2021-03-07 10:00:00-08:00 <----
2021-03-14 10:00:00-07:00 <----
2021-03-21 10:00:00-07:00
2021-03-28 10:00:00-07:00
Sad thing is @xjlin0 s example doesn't work when retrieving occurrences via a Period:
import pytz
import datetime
from django.utils import timezone
from schedule.models import Event, Occurrence, Calendar
from schedule.models.rules import Rule
import schedule
utc = timezone.utc
pacific = pytz.timezone('America/Los_Angeles')
start = utc.localize(datetime.datetime(2020, 1, 12, 18, 0, 0))
end = utc.localize(datetime.datetime(2020, 1, 12, 20, 0, 0)) #Janurary is PST
rule = Rule(
name="test_rule",
description="WEEKLY",
frequency="WEEKLY"
)
rule.save()
calendar = Calendar(
name="test_calendar"
)
calendar.save()
event = Event(
title="test_event",
calendar=calendar,
start=start,
end=end,
rule=rule
)
event.save()
o_start = pacific.localize(datetime.datetime(2021, 3, 1, 0, 0, 0))
o_end = pacific.localize(datetime.datetime(2021, 3, 31, 0, 0, 0)) # Daylight starts in Mar.14'21
request = None
event_list = schedule.settings.GET_EVENTS_FUNC(request, calendar)
period = schedule.periods.Period(
event_list,
o_start,
o_end
)
occs = period.get_occurrences() # query with local time!!!
for occ in occs:
print(occ.start.astimezone(pacific))
This outputs:
2021-03-07 10:00:00-08:00 <----
2021-03-14 11:00:00-07:00 <----
2021-03-21 11:00:00-07:00
2021-03-28 11:00:00-07:00
Just double confirm that @Lucianovici 's solution of adding tzinfo works across day light saving days.
period = Period(events, start_date, end_date, tzinfo=start_date.tzinfo)
return period.get_occurrences()
https://github.com/llazzaro/django-scheduler/issues/304#issuecomment-348757970