Statistics page "Starting at" data / column incorrect

[towster]

Is there an existing issue for this?

• [X] I have searched the existing issues

What happened?

Using drives as an example but goes for charges as well. I took screenshot of time range "last 2 days" at 8:20pm CDT.

First off timezone. Every time I goto statistics page it resets the timezone to Berlin. My browser and My profile have it set to Chicago.

Next we have when a day starts. Current time of screenshot is 8:20pm. I have selected day view and last two days. Shows me 3 days. The first day is the 18th (tomorrow!) which apparently starts at 7pm today (the 17th). It shows I have driven 3 places in the past 1 hour and 20 minutes. The lefthand column in the screenshot of the dates/times are my drives for the last 2 days. My most recent drive was 4/17 @ 6:07 (for 6 minutes) or 2 hours and 7 minutes ago. None of these drives are after the cutoff time shown.

Expected Behavior

timezone should be set properly when going to page

the time that a day should start should be midnight. Count of number of drives should match the number of drives for that day as shown in drives page.

Steps To Reproduce

Goto statistics page change time period to days set time range to last 2 days

Relevant log output

teslamate-1  | 2024-04-17 20:34:31.270 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/3744882384725593848/vehicle_data -> 408 (9178.959 ms)
teslamate-1  | 2024-04-17 20:34:31.271 [warning] TeslaApi.Error / %{"error" => "{\"error\": \"timeout\"}", "error_description" => "", "response" => nil}
teslamate-1  | 2024-04-17 20:34:31.271 car_id=1 [error] Error / :unknown
teslamate-1  | 2024-04-17 20:36:08.682 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/3744882384725593848/vehicle_data -> 408 (9143.538 ms)
teslamate-1  | 2024-04-17 20:36:08.682 [warning] TeslaApi.Error / %{"error" => "{\"error\": \"timeout\"}", "error_description" => "", "response" => nil}
teslamate-1  | 2024-04-17 20:36:08.682 car_id=1 [error] Error / :unknown
teslamate-1  | 2024-04-17 20:36:37.808 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/3744882384725593848/vehicle_data -> 408 (9124.815 ms)
teslamate-1  | 2024-04-17 20:36:37.808 [warning] TeslaApi.Error / %{"error" => "{\"error\": \"timeout\"}", "error_description" => "", "response" => nil}
teslamate-1  | 2024-04-17 20:36:37.809 car_id=1 [error] Error / :unknown
teslamate-1  | 2024-04-17 20:37:06.946 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/3744882384725593848/vehicle_data -> 408 (9136.254 ms)
teslamate-1  | 2024-04-17 20:37:06.946 [warning] TeslaApi.Error / %{"error" => "{\"error\": \"timeout\"}", "error_description" => "", "response" => nil}
teslamate-1  | 2024-04-17 20:37:06.946 car_id=1 [error] Error / :unknown
teslamate-1  | 2024-04-17 20:38:58.978 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/3744882384725593848/vehicle_data -> 408 (9130.891 ms)
teslamate-1  | 2024-04-17 20:38:58.979 [warning] TeslaApi.Error / %{"error" => "{\"error\": \"timeout\"}", "error_description" => "", "response" => nil}
teslamate-1  | 2024-04-17 20:38:58.979 car_id=1 [error] Error / :unknown
teslamate-1  | 2024-04-17 20:39:28.125 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/3744882384725593848/vehicle_data -> 408 (9145.764 ms)
teslamate-1  | 2024-04-17 20:39:28.125 [warning] TeslaApi.Error / %{"error" => "{\"error\": \"timeout\"}", "error_description" => "", "response" => nil}
teslamate-1  | 2024-04-17 20:39:28.125 car_id=1 [error] Error / :unknown
teslamate-1  | 2024-04-17 20:39:57.273 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/3744882384725593848/vehicle_data -> 408 (9146.948 ms)
teslamate-1  | 2024-04-17 20:39:57.274 [warning] TeslaApi.Error / %{"error" => "{\"error\": \"timeout\"}", "error_description" => "", "response" => nil}
teslamate-1  | 2024-04-17 20:39:57.274 car_id=1 [error] Error / :unknown
teslamate-1  | 2024-04-17 20:40:26.409 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/3744882384725593848/vehicle_data -> 408 (9134.388 ms)
teslamate-1  | 2024-04-17 20:40:26.409 [warning] TeslaApi.Error / %{"error" => "{\"error\": \"timeout\"}", "error_description" => "", "response" => nil}
teslamate-1  | 2024-04-17 20:40:26.409 car_id=1 [error] Error / :unknown
teslamate-1  | 2024-04-17 20:42:02.156 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/3744882384725593848/vehicle_data -> 408 (9159.892 ms)
teslamate-1  | 2024-04-17 20:42:02.156 [warning] TeslaApi.Error / %{"error" => "{\"error\": \"timeout\"}", "error_description" => "", "response" => nil}
teslamate-1  | 2024-04-17 20:42:02.157 car_id=1 [error] Error / :unknown
teslamate-1  | 2024-04-17 20:42:31.301 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/3744882384725593848/vehicle_data -> 408 (9143.539 ms)
teslamate-1  | 2024-04-17 20:42:31.301 [warning] TeslaApi.Error / %{"error" => "{\"error\": \"timeout\"}", "error_description" => "", "response" => nil}
teslamate-1  | 2024-04-17 20:42:31.301 car_id=1 [error] Error / :unknown
teslamate-1  | 2024-04-17 20:43:00.427 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/3744882384725593848/vehicle_data -> 408 (9124.103 ms)
teslamate-1  | 2024-04-17 20:43:00.427 [warning] TeslaApi.Error / %{"error" => "{\"error\": \"timeout\"}", "error_description" => "", "response" => nil}
teslamate-1  | 2024-04-17 20:43:00.427 car_id=1 [error] Error / :unknown
teslamate-1  | 2024-04-17 20:43:12.974 [info] GET /
teslamate-1  | 2024-04-17 20:43:12.978 [info] Sent 200 in 4ms
teslamate-1  | 2024-04-17 20:43:17.813 [info] GET /settings
teslamate-1  | 2024-04-17 20:43:17.822 [info] Sent 200 in 9ms
teslamate-1  | 2024-04-17 20:43:29.601 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/3744882384725593848/vehicle_data -> 408 (9172.466 ms)
teslamate-1  | 2024-04-17 20:43:29.601 [warning] TeslaApi.Error / %{"error" => "{\"error\": \"timeout\"}", "error_description" => "", "response" => nil}
teslamate-1  | 2024-04-17 20:43:29.601 car_id=1 [error] Error / :unknown
teslamate-1  | 2024-04-17 20:43:58.720 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/3744882384725593848/vehicle_data -> 408 (9117.199 ms)
teslamate-1  | 2024-04-17 20:43:58.720 [warning] TeslaApi.Error / %{"error" => "{\"error\": \"timeout\"}", "error_description" => "", "response" => nil}
teslamate-1  | 2024-04-17 20:43:58.720 car_id=1 [error] Error / :unknown
teslamate-1  | 2024-04-17 20:44:27.867 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/3744882384725593848/vehicle_data -> 408 (9145.749 ms)
teslamate-1  | 2024-04-17 20:44:27.867 [warning] TeslaApi.Error / %{"error" => "{\"error\": \"timeout\"}", "error_description" => "", "response" => nil}
teslamate-1  | 2024-04-17 20:44:27.867 car_id=1 [error] Error / :unknown
teslamate-1  | 2024-04-17 20:44:57.021 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/3744882384725593848/vehicle_data -> 408 (9152.151 ms)
teslamate-1  | 2024-04-17 20:44:57.021 [warning] TeslaApi.Error / %{"error" => "{\"error\": \"timeout\"}", "error_description" => "", "response" => nil}
teslamate-1  | 2024-04-17 20:44:57.021 car_id=1 [error] Error / :unknown
teslamate-1  | 2024-04-17 20:44:59.346 [info] GET /
teslamate-1  | 2024-04-17 20:44:59.350 [info] Sent 200 in 4ms
teslamate-1  | 2024-04-17 20:45:26.163 [info] GET https://owner-api.teslamotors.com/api/1/vehicles/3744882384725593848/vehicle_data -> 408 (9141.873 ms)
teslamate-1  | 2024-04-17 20:45:26.163 [warning] TeslaApi.Error / %{"error" => "{\"error\": \"timeout\"}", "error_description" => "", "response" => nil}
teslamate-1  | 2024-04-17 20:45:26.163 car_id=1 [error] Error / :unknown

Screenshots

No response

Additional data

No response

Type of installation

Docker

Version

1.28.5

[DrMichael]

What happened?

First off timezone. Every time I goto statistics page it resets the timezone to Berlin. My browser and My profile have it set to Chicago.

Interesting. Can you please look, which timezone the database has? docker compose exec database psql -U teslamate -d teslamate Then SELECT current_setting('TIMEZONE');

(BTW, the logs are not helping, they just say, that your teslamate could not connect to the tesla servers. That might happen.)

[cwanja]

Edited comment to clear up quoted messages.

[towster]

Yeah i know the logs didnt help but it was a required field.. thx for confirming at least that it is normal for tesla to have server down.. ;)

WARN[0000] /root/docks/teslamate/docker-compose.yml: `version` is obsolete 
psql (15.6 (Debian 15.6-1.pgdg120+2))
Type "help" for help.

teslamate=# SELECT current_setting('TIMEZONE');
 current_setting 
-----------------
 Etc/UTC
(1 row)

teslamate=#

Hasnt been asked for but.. I only added the TZ setting to the other containers after i saw it was having this issue.. it didnt change anything..

version: "3"

services:
  teslamate:
    image: teslamate/teslamate:latest
    restart: always
    environment:
      - ENCRYPTION_KEY=___NOT_MY_PASS___ #replace with a secure key to encrypt your Tesla API tokens
      - DATABASE_USER=teslamate
      - DATABASE_PASS=___NOT_MY_PASS___ #insert your secure database password!
      - DATABASE_NAME=teslamate
      - DATABASE_HOST=database
      - MQTT_HOST=mosquitto
      - TZ=America/Chicago
    ports:
      - 4000:4000
    volumes:
      - ./import:/opt/app/import
    cap_drop:
      - all

  database:
    image: postgres:15
    restart: always
    environment:
      - POSTGRES_USER=teslamate
      - POSTGRES_PASSWORD=___NOT_MY_PASS___ #insert your secure database password!
      - POSTGRES_DB=teslamate
      - TZ=America/Chicago
    volumes:
      - teslamate-db:/var/lib/postgresql/data

  grafana:
    image: teslamate/grafana:latest
    restart: always
    environment:
      - DATABASE_USER=teslamate
      - DATABASE_PASS=___NOT_MY_PASS___ #insert your secure database password!
      - DATABASE_NAME=teslamate
      - DATABASE_HOST=database
      - TZ=America/Chicago
    ports:
      - 3000:3000
    volumes:
      - teslamate-grafana-data:/var/lib/grafana

  mosquitto:
    image: eclipse-mosquitto:2
    restart: always
    command: mosquitto -c /mosquitto-no-auth.conf
    # ports:
    #   - 1883:1883
    volumes:
      - mosquitto-conf:/mosquitto/config
      - mosquitto-data:/mosquitto/data

volumes:
  teslamate-db:
  teslamate-grafana-data:
  mosquitto-conf:
  mosquitto-data:

[DrMichael]

And with that settings you get Etc/UTC in the database? Does not explain Europe/Berlin though...

[cwanja]

And with that settings you get Etc/UTC in the database? Does not explain Europe/Berlin though...

I looked through the Dashboard JSON and you can see Berlin in the code but it's how Adrian deployed it over two years ago (if I am following the Grafana JSON correctly). I am not a heavy user of the dashboard, so not sure if it's always been that way.

[towster]

Are either of you able to duplicate what I am seeing? Should I delete anything and recreate it? (I would rather not lose my historical data tho)

[cwanja]

Are either of you able to duplicate what I am seeing? Should I delete anything and recreate it? (I would rather not lose my historical data tho)

First issue

First off timezone. Every time I goto statistics page it resets the timezone to Berlin. My browser and My profile have it set to Chicago.

I just reloaded this dashboard from TeslaMate and it loaded America/Chicago just fine which coresponds to my TZ variable in my YML file. However, if I load from the "Dashboards" menu on the right side of dashboards, it does not work. @towster can you see the same behavior in the two tests? If so, I would say this is a separate issue.

Second issue

Next we have when a day starts. Current time of screenshot is 8:20pm. I have selected day view and last two days. Shows me 3 days. The first day is the 18th (tomorrow!) which apparently starts at 7pm today (the 17th). It shows I have driven 3 places in the past 1 hour and 20 minutes. The lefthand column in the screenshot of the dates/times are my drives for the last 2 days. My most recent drive was 4/17 @ 6:07 (for 6 minutes) or 2 hours and 7 minutes ago. None of these drives are after the cutoff time shown.

I recall being able to see this last night, but I was in the middle of testing PRs. Back on master release, I do not see the issue both in using the browser time to select last two days or physically selecting two days in the calendar view.

Edit: now that I look back at the screenshot, the "Start at" date looks to be in UTC and not local time.

[towster]

Admin user's profile - timezone set to Chicago:

From overview i click dropdown on right side for dashboard/statistics and I get this:

Here is the settings for the statistics dashboard itself

How do I reload the dashboard to compare?

[cwanja]

You need to set the TZ in the docker-compose, I believe that is how TeslaMate is passing it into the Statistics URL from TeslaMate homepage. And if so, we need to update the Dashboards dropdown across dashboards to use this same variable.

[cwanja]

https://github.com/teslamate-org/teslamate/blob/c833a30323f6d014f56e4b9a86e0be9ca8d7f8b1/assets/js/main.js#L22 here is where the Statistics from TeslaMate home is passing the timezone into the URL.

[towster]

You need to set the TZ in the docker-compose, I believe that is how TeslaMate is passing it into the Statistics URL from TeslaMate homepage. And if so, we need to update the Dashboards dropdown across dashboards to use this same variable.

So set the TZ variable in docker-compose - delete the grafana docker - delete the grafana volume - compose pull - compose up -d

Or is this wrong? Or a simpler way?

[cwanja]

Do not believe you have to delete the Grafana docker and volume. Just docker compose down; docker compose up -d. You can immediately check that by hovering over the Statistics URL from TeslaMate homepage.

[towster]

Ok did compose down / pull / up -d... Went to teslamate gui (and not grafana).. moused over the statistics dashboard link.. it does pass the chicago timezone on the URL.. and when clicked on the statistics page does come up with being in chicago rather than berlin.. I guess that helps but seems to me the statistics should properly read my TZ rather than force me to leave grafana and go back to teslamate just to view that report..

And then when you switch to day view it still shows the days starting at 7pm..

[DrMichael]

Ok, time to test... I took out all the timezone magic and the timezone variable. Since Grafana 10.1.0 (https://github.com/grafana/grafana/pull/66785) there is a __timezone macro (I asked for that in 2020 https://github.com/grafana/grafana/discussions/38159), which makes the defined timezone available to the queries.

Please test. Looks good for me, but could look different with larger distant from UTC. Statistics-New.json

[JakobLichterfeld]

Please test. Looks good for me, but could look different with larger distant from UTC. Statistics-New.json

Please open a PR for every change, as it is a transparent way, and our CI runs can build dedicated images per PR for testing purposes, no need to pass file this way :-) (see https://docs.teslamate.org/docs/development/#testing-with-our-ci-which-builds-the-docker-images-automatically-per-pr)

[DrMichael]

Oh, @JakobLichterfeld I really appreciate your role as guardian... I rather don't want to propose a change, which does not resolve the issues here. So, would like to have it tested with timezones with big difference to UTC... But will make a pull request on the weekend, ok?

[cwanja]

@towster I am going to edit your issue and split it to focus just on the dashboard data. Regarding links and the environment variable, that is going to be focused in #3858.

[JakobLichterfeld]

Oh, @JakobLichterfeld I really appreciate your role as guardian... I rather don't want to propose a change, which does not resolve the issues here. So, would like to have it tested with timezones with big difference to UTC... But will make a pull request on the weekend, ok?

I take this as a compliment, ;-) Transparent way would be a pull request marked as work in progress. So it's clear it is not ready yet to be merges, but your change is transparent and CI is running to give others the ability to test with no hassle.

[towster]

I imported the statistics-new.json dashboard. When it is opened it no longer says berlin (yay). The data in it is still odd though. When I change it to day timeframe it shows I have 4 drives after 7pm yesterday. When I click on the "4" to see the drives the drives page only displays 2 drives..

[DrMichael]

Oh, @JakobLichterfeld I really appreciate your role as guardian...

I take this as a compliment, ;-)

It was honestly meant as such!

[DrMichael]

I imported the statistics-new.json dashboard. When it is opened it no longer says berlin (yay). The data in it is still odd though. When I change it to day timeframe it shows I have 4 drives after 7pm yesterday. When I click on the "4" to see the drives the drives page only displays 2 drives..

So, I created a PR https://github.com/teslamate-org/teslamate/pull/3862. @towster can you please try that?

I am still wondering, why it says Starting at ...07:00...

[towster]

I tried PR #3862 but there are still issues.

First off I have not driven anywhere today 4/20. It says I have driven 8 places on 4/20. I did drive 8 places on 4/19 though.

If I click on the "8" to see the 8 drives from 4/20 I get only 6 drives and some of them are from 4/18 and some are from 4/19. The timeframe that is selected on the drive page is 4/18@19:00-4/19@19:00.

What statistics.json shows.

What drives shows for past 2 days.

What shows when I click on the 8 drives for 4/20.

[DrMichael]

@towster you have a great test scenario... My difference to UTC is too small to be significant. I made a change and committed it to the PR. You you pull and retry? @JakobLichterfeld image: ghcr.io/teslamate-org/teslamate/grafana:pr-3862 does not appear to reflect the change?!? Any idea?

[JakobLichterfeld]

@JakobLichterfeld image: ghcr.io/teslamate-org/teslamate/grafana:pr-3862 does not appear to reflect the change?!? Any idea?

Not really, tested it on my side as well, and your changes seems to be not correctly reflected in the Grafana image. Currently, I'm out of time to dig deeper.

[towster]

First off I think I am doing something wrong with the docker compose and the pr - it looked like it got it but when things come up statistics still shows the one with the TZ selection..

Regardless I browsed into the PR and downloaded the json and then imported the dashboard. So on the monthly view something is broke.. it shows days.. sorta.. but it is set on monthly..

So i switched to day view.. It still shows things starting at 7pm..

Here I am clicking on the -- drives next to the 4/20 date.. and I get 6 drives..(time range 4/18@1900-4/19@1900)

[DrMichael]

Sorry, I was focused on charges and day view (the only case, where it makes a difference for me). Try new version please https://github.com/DrMichael/teslamate/blob/master/grafana/dashboards/statistics.json Ignore the links for now. Please compare Statistics to Drives/Charges.

[towster]

Getting closer.. Month view is fixed again.. Switched to days and it shows midnite to midnite..

I click on any of the links within the page they all pull up time frame from 7pm to 7pm.. for example I click on the "--" under drives on 4/20.. and I get 4 drives.. all 4 actually on 4/19 after 7pm.. Since I did not drive anywhere on 4/20 I would expect to not see any results..

PS - sorry i just re-read your post and realized i looked at links too -- i think the text on the stats page is correct now - just the link is wrong..

[DrMichael]

Ok, new version, pls give it a try. https://github.com/DrMichael/teslamate/blob/master/grafana/dashboards/statistics.json (I switched my browser to your timezone. I get strange midnights in March at 23.00. Not sure why... Might be the different start dates of DST...)

[towster]

Sorry i havent tested anything last couple days. I just looked at it and I think it fixed everything that I have mentioned. I will play with it more and see if i can find anything but I think it is good to go..