Issues with GTFS timezones

[kjhovi]

The problem

When using the GTFS integration (with http://karttapalvelu.kuopio.fi/google_transit/google_transit.zip), I get departure times offset by the same amount as my timezone.

So say I have a departure time at 16:20, the sensor would show that to me as 18:20. If I change my timezone(Europe/Helsinki (UTC +2)) in the configuration to UTC, the departure times are shown correctly.

I guess this has to do with the recently added timezone code
self._state = self._departure["departure_time"].replace(tzinfo=dt_util.UTC)

It seems to me, that the code just defines whatever is set in the departure_time as UTC, and then the UI accounts for this by offsetting it by 2h. In my case the gtfs data is defined in local times, and the data has the agency timezones defined as "Europe/Helsinki", so I guess the integration does not take that into account at the moment. So my departure time seems correct until it is set to UTC in the line mentioned above.

What version of Home Assistant Core has the issue?

core-2021.12.8

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Core

Integration causing the issue

GTFS

Link to integration documentation on our website

https://www.home-assistant.io/integrations/gtfs/

Example YAML snippet

sensor:
  - platform: gtfs
    origin: 201447
    destination: 303022
    data: vilkku
    name: Bussi 5
    include_tomorrow: true

Anything in the logs that might be useful for us?

No response

Additional information

The gtfs data Link I provided had some issues with some weird characters in the beginning of the files agency.txt and routes.txt (and maybe feed_info.txt). Removing the weirdness from the beginning of the file created the sqlite database correctly.

[probot-home-assistant[bot]]

gtfs documentation gtfs source _{^{(message by IssueLinks)}}

[pepoweb]

+1 Same issue in Tallinn

[Friedrieck]

Same issue in Europe/Brussels TZ

[Kimble8650]

I've read a bit into the source code of both the GTFS integration and the datetime library and I think the issue might be solved with appending the agency timezone value in a "+02:00" -format for UTC+2 (as specified by ISO 8601 https://www.progress.com/blogs/understanding-iso-8601-date-and-time-format) to the line 424

origin_depart_time = f"{now_date} {item['origin_depart_time']}"

Then it would be a matter of finding out the agency timezone before that line of code. So the hypothetical code would be something like origin_depart_time = f"{now_date} {item['origin_depart_time']}{agency_timezone_code}" If the agency timezone proves to be too difficult of a task to get in code, an improvement would even be to have an optional configuration to specify the timezone manually.

Then the dt_util would (probably) parse the string to a datetime and include the correct timezone specified on line 442 depart_time = dt_util.parse_datetime(origin_depart_time)

This is all just guesswork, as I don't have the means to actually try this myself. Any thoughts?

[Friedrieck]

With the GTFS I use, I see this in the attributes of the entity: "Agency agency timezone Europe/Brussels", so it seems available. Also to note, there are a lot of date/time attributes in the entity and they all show the time correctly. Only the main date/time, i.e. the state, is shown wrong.

[Kimble8650]

@Friedrieck yes, in the code the agency timezone just seems to be resolved some time after the departure time is resolved. So there might be an order-of-operations type of issue here.

I have tried to make a hack by using a template sensor, that reads the attributes, that are correct, but so far I have not been able to get the same nice timestamp formatting for the templates (without making a much more complex template) as with the orig. sensor (The 'in 5 minutes' is the formatting I'm looking for).

[Friedrieck]

@Kimble8650, the formatting should not be a concern to you, this is taken care of by the frontend. For instance, my GTFS sensor currently displays "Dans 2 heures" (in two hours); when you open the more-info dialog, it shows "9 février 2022, 16:15", but you see the actual value in the dev tools, it is: 2022-02-09T15:15:00+00:00

The real strange thing is that there is one attribute, arrival, that shows '9 février 2022, 15:39:00' in the more-info dialog, but of which the state is 2022-02-09T14:39:00+00:00 in dev tools. So here, the correct timezone is applied, while for the state, it is not.

[Friedrieck]

From what I see in the dev tools, and assuming the display of the state and the attributes in the UI have the same logic, I think that if there was no TZ info in the entity's state value, it would show correctly, i.e. 2022-02-09T15:15:00 instead of 2022-02-09T15:15:00+00:00 (15:15 is the correct time in my TZ)

[Friedrieck]

get the same nice timestamp formatting for the templates

So , provide it with a well-formatted timestamp, and specify the device_class: timestamp

[Kimble8650]

Thanks @Friedrieck! I was at first dishearted by my earlier attempts, but decided to try again. And low and behold, I finally managed to get it working! Below is my config: - platform: template sensors: bus_departure_hack: device_class: timestamp value_template: "{{state_attr('sensor.bus_no5', 'origin_stop_departure_time') | as_timestamp | timestamp_local }}"

I was a bit surprised when browsing the forums, that this behaviour does not seem to be too well known and/or documented, as I found multiple threads where people tried to accomplish this exact thing, and the only solution they were provided with was to write a complex string template to manually squeeze out the "in 5 minutes" -string.

Still, this is less than optimal, and I'd prefer using only the sensor I defined for this specific purpose instead of this hack.

[Friedrieck]

@frenck , sorry to @mention you like that, but I think your recent changes on the formatting of the timestamp value for the state of the sensor may have introduced a bug. The time of the state as it is displayed now in the UI is wrong (it may look correct if you are in GMT TZ...), while all the other time values in the attributes are correct, such as the arrival attribute. It looks as if your change just forces the time value to be seen as UTC, whatever the actual source data. I'm not really a dev, but based on the fact that the value for the arrival attribute is correct, it seems that instead of this (line 616): self._state = self._departure["departure_time"].replace(tzinfo=dt_util.UTC) It should be this: self._state = dt_util.as_utc(self._departure["departure_time"]).isoformat() As on line 668.

From what I see of the other date/time attributes, they seem to be rendered as they are stored in the source DB, i.e. probably already in the agency's timezone, and without the TZ info in the field.

[Friedrieck]

Dear devs, It seems that since this change: e0110737897154e2fa9b7086d5d787681ff86e16 (not sure what it was supposed to fix), the following attempt to fix the damaged state timestamp TZ has not worked (841e22258de491eeaa7f578446104cef029baeab, see above thread). From the correct behaviour observed on the arrival attribute, it seems the change introduced on 8 Dec. last should actually be rolled back.

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[Kimble8650]

This is still very much a problem, and has not been fixed in the latest versions.

[theocoutu]

I've been experiencing the same problem, the 'next bus' time is the time in GMT, not the local time zone.

[vingerha]

Not sure if this helps raising attention, same problem here, especially since it is stored as a timezone, it shows 2 hours behind (I am in CET)

[simgaymer]

I too am encountering the exact same issue in the United States Central Standard Time.

[puzzledsam]

Also experiencing this issue, which makes it impractical to use the bus times as glanceable information.

[vingerha]

Is this still supported? Issue reported about 1 year ago and no action?

[Kimble8650]

Is this still supported? Issue reported about 1 year ago and no action?

Yeah, I was wondering the same thing. I mean, I do get that there are a LOT of input on potential issues, so not all of them will be prioritised to be addressed in the near future. But considering that, to my understanding, this issue was introduced in a fix not long before this issue was reported, I feel it's weird that nothing has been even communicated regarding this.

Anyway, I'm still hoping on this getting fixed at some point, since it'd really make my life a lot easier =)

[vingerha]

I have been diving a bit into it and I have a few ideas:

1. simplest: if agency_timezone exists then format along that one, else use UTC
2. less easy: allow to define the TZ for the sensors, per sensor in the config and maybe an order config > agency > UTC

Why not taking HA timezone? Because it depends on the gtfs data which may or may not represent current TZ

Any feedback appreciated so I can prepare a PR

[Kimble8650]

Great to have someone finally looking at fixing this! <3 How I think this ought to be handled in code is, that we handle all the calculations in UTC, and when we have a final answer on a departure/arrival time, we display it in the HA timezone.

So we would use the agency timezone to convert the schedules to UTC, do the calculations, and then display the results in the HA timezone. That way, if we have an edge case where the user lives on the border of two timezones (eg. Finland & Sweden), and s/he would live in Finland (UTC+2), but often commute in Sweden (Agency timezone = UTC +1), the user would know to depart at the right time.

Am I making sense? I feel like I'm not as good at explaining this, as I ought to be :D

[vingerha]

@Kimble8650 I believe it is the same as I wrote but I indeed forgot to document the whole picture as I read/see it, in the end repeating the options

Issue: the gtfs sensor(s) 'state' value shows a timezone format but that does not take into account the timezone of the data itself, where it always applies UTC This is also shown for the attribute 'arrival' (on the top of attributes list) Other attributes working with arrival/departure time are OK as they are not imezone formatted

Possible sources of timezone

1. gtfs data set, attribute of 'agency'
2. manual, i.e. part of the gtfs sensor config
3. home assistant (server)

Proposed solutions to both (!) sensor-state and attribute: arrival

1. simplest: check if agency_timezone exists, then format along that one, else use UTC 2 less easy: manually define the TZ for the sensors, per sensor. Login: if in config then that, else check if agency)tz exists else use UTC

Note that I prefer 1 as it is in the code and less risky on delivery as I am the only tester.

Awaiting others with ideas or confirmation

[Kimble8650]

Ah. I see! Honestly, if I'm being selfish here, any of the solutions work as long as the solution takes daylight saving into account otherwise the solution only works half the time =). I don't know how exactly the timestamps are handled, but I would guess that if using datetimes properly, those are handled out-of-the-box in python.

Again, I really appreciate, that you are looking into this!

[pepoweb]

@vingerha I gladly help out testing.

As for the solutions, I think keeping it in the code would be preferred.

[simgaymer]

I'd be happy to help out with testing as well! Glad to see some progress on this issue as I'm loosing hairs programming workaround templates.

[vingerha]

Thanks, I have a first version here and being a not-developer, I found not other way to share this then via customer component. Also travelling so not responding fast Doc is minimal but I hope you get the idea on how to add to HACS or folder https://github.com/vingerha/gtfs2

Observation...although I use the very same TZ method, for the state it takes the +00:00 (but that is then +1h so correct with me) For attribute it takes +01:00 ... no clue why the diff (see: being not-developer) This works for me in CET, I have Europe/Paris in table agency field agency_timezone

[simgaymer]

My word... @vingerha, you've done it!! All times are showing correct on my dashboard!!

[vingerha]

Good... before I create the PR (later this week) I'd like to at least have @pepoweb feedback too. My self I am happy of course as I worked on it until it fixed my issue :)

[Kimble8650]

This sounds really promising! Could you try what happens if you go to Settings -> System -> General and change the timezone to something else? Do the times change as well? Or is the integration actually smart enough to calculate a new departure/arrival to accommodate the new timezone? I'm just curious to find the edge cases =).

Btw, is do you know of any good guides to actually start collaborating and testing stuff in home assistant? I'm really curious to help out in this community effort, but what I'd need is a 'for dummies' type of guide to get started =). (I am a developer by profession, but setting up a completely new environment is always a hassle :) )