teslamate icon indicating copy to clipboard operation
teslamate copied to clipboard

Update statistics.json: timezone issues addressed

Open DrMichael opened this issue 10 months ago • 14 comments

Change to use $__timezone macro.

DrMichael avatar Apr 20 '24 09:04 DrMichael

Deploy Preview for teslamate ready!

Name Link
Latest commit ffd8ba7a59e9eb301fb66294027820818dc98f63
Latest deploy log https://app.netlify.com/sites/teslamate/deploys/6627b3ab2dbbc5000834ae99
Deploy Preview https://deploy-preview-3862--teslamate.netlify.app
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify[bot] avatar Apr 20 '24 09:04 netlify[bot]

To test this changes, we can update our docker-compose.yml as follows (see https://docs.teslamate.org/docs/development#testing-with-our-ci-which-builds-the-docker-images-automatically-per-pr)

grafana:
       # image: teslamate/grafana:latest
      image: ghcr.io/teslamate-org/teslamate/grafana:pr-3862

do a docker compose pull followed by a docker compose up -d afterwards

JakobLichterfeld avatar Apr 20 '24 11:04 JakobLichterfeld

To test this changes, we can update our docker-compose.yml as follows (see https://docs.teslamate.org/docs/development#testing-with-our-ci-which-builds-the-docker-images-automatically-per-pr)


grafana:

       # image: teslamate/grafana:latest

      image: ghcr.io/teslamate-org/teslamate/grafana:pr-3862

do a docker compose pull followed by a docker compose up -d afterwards

Going to ask a dumb question here (forgive me). How would you remove these 'test' images once you are done testing? Or does it not retain them somehow? Mainly thinking space on my Pi.

cwanja avatar Apr 20 '24 11:04 cwanja

Wondering if "timezone" in all of our dashboard JSONs should be "browser". https://github.com/search?q=repo%3Ateslamate-org%2Fteslamate%20timezone&type=code

cwanja avatar Apr 20 '24 11:04 cwanja

This does not work for me. Or I am not pulling the image correctly 🤷🏻‍♂️ I continue to see the timezone dropdown. Looking through the JSON, I do not see the iteration to verify it was upped. Followed the guidance above:

  grafana:
#    image: teslamate/grafana:latest
    image: ghcr.io/teslamate-org/teslamate/grafana:pr-3862

But still not working. Tried to prune all of my docker images with docker image prune -a and docker container prune to no luck.

cwanja avatar Apr 20 '24 12:04 cwanja

How would you remove these 'test' images once you are done testing? Or does it not retain them somehow? Mainly thinking space on my Pi.

Good point. There are several ways, but if you want to remove dangling images, this would be safest imo:

docker image prune -a

WARNING! This will remove all images without at least one container associated to them.

see https://docs.docker.com/reference/cli/docker/image/prune/

JakobLichterfeld avatar Apr 20 '24 12:04 JakobLichterfeld

This needs to be tested still... Edge cases are drives and charges between midnight UTC and midnight local time.

DrMichael avatar Apr 20 '24 12:04 DrMichael

Wondering if "timezone" in all of our dashboard JSONs should be "browser". https://github.com/search?q=repo%3Ateslamate-org%2Fteslamate%20timezone&type=code

AFAICS one is browser, the others are empty, probably also meaning browser.

DrMichael avatar Apr 20 '24 12:04 DrMichael

I am a bit struggling here... I have two charges: in the database:

 id  |       start_date
-----+-------------------------
 576 | 2024-01-20 21:37:50.375
 577 | 2024-01-20 23:01:09.237
(2 rows)

Since it is UTC and my local time is Europe/Berlin, the second was indeed on 2024-01-21 00:01 and should show up for the 2024-01-21, but it show up for 2024-01-20.

Relevant query:

WITH data AS (
  SELECT
    charging_processes.*,
    date_trunc('$period', TIMEZONE('$__timezone', start_date)) as local_period
    FROM charging_processes)
SELECT
  EXTRACT(EPOCH FROM date_trunc('$period', TIMEZONE('$__timezone', local_period)))*1000 AS date_from,
  EXTRACT(EPOCH FROM date_trunc('$period', TIMEZONE('$__timezone', local_period) + ('1 ' || '$period')::INTERVAL))*1000 AS date_to,
  CASE '$period'
    WHEN 'month' THEN to_char(local_period, 'YYYY Month')
    WHEN 'year' THEN to_char(local_period, 'YYYY')
    WHEN 'week' THEN 'week ' || to_char(local_period, 'WW') || ' starting ' || to_char(local_period, 'YYYY-MM-DD')
    ELSE to_char(local_period, 'YYYY-MM-DD')
  END AS display,
  local_period AS date,
  sum(greatest(charge_energy_added,charge_energy_used)) AS sum_consumption_kwh,
  sum(greatest(charge_energy_added,charge_energy_used)) / count(*) AS avg_consumption_kwh,
  sum(cost) AS cost_charges,
  count(*) AS cnt_charges
FROM data WHERE
  car_id = $car_id AND
  $__timeFilter(start_date) AND
  (charge_energy_added IS NULL OR charge_energy_added > 0.1)
GROUP BY date
ORDER BY date

I tried to apply the timezone correction to the other, but it did not help. Any thoughts?

DrMichael avatar Apr 20 '24 14:04 DrMichael

This does not work for me. Or I am not pulling the image correctly 🤷🏻‍♂️ I continue to see the timezone dropdown. Looking through the JSON, I do not see the iteration to verify it was upped.

Same here. But the images have been produced...

DrMichael avatar Apr 21 '24 09:04 DrMichael

This does not work for me. Or I am not pulling the image correctly 🤷🏻‍♂️ I continue to see the timezone dropdown. Looking through the JSON, I do not see the iteration to verify it was upped.

Same here. But the images have been produced...

@JakobLichterfeld is there something we are missing? Does it go in Grafana or TeslaMate service?

cwanja avatar Apr 21 '24 13:04 cwanja

The change is only in Grafana...

DrMichael avatar Apr 21 '24 13:04 DrMichael

@JakobLichterfeld is there something we are missing? Does it go in Grafana or TeslaMate service?

Not really. It must be included in the grafana image. It on my side as well, and your changes seems to be not correctly reflected in the Grafana image. Currently, I'm out of time to dig deeper. (https://github.com/teslamate-org/teslamate/issues/3857#issuecomment-2068640395)

JakobLichterfeld avatar Apr 22 '24 07:04 JakobLichterfeld

I'd like some more testing from users with significant timezone difference to UTC, e.g. UTC+-5 or more...

DrMichael avatar Apr 25 '24 08:04 DrMichael

I'd like some more testing from users with significant timezone difference to UTC, e.g. UTC+-5 or more...

UTC -5 (or 6 depending on the time of year) hours and just tested.

  1. Timezone dropdown is removed.
  2. Verified that it starts at 12AM in month and day view.
  3. Month view looks correct.
  4. Clicking on a day, the timeframe is midnight to midnight of the selected day.

One item I did find, testing a use-case from @towster - it seems the Drives dashboard filters out (either) zero duration or zero distance drives. I saw this when comparing the filtered Drives dashboard versus the same day Trip dashboard links off the Statistic page. This might be by design.

cwanja avatar May 02 '24 02:05 cwanja

I am approving and merging. Linking all issues such that they are closed when merged.

Great work @DrMichael and thank you 😄

cwanja avatar May 02 '24 02:05 cwanja

One item I did find, testing a use-case from @towster - it seems the Drives dashboard filters out (either) zero duration or zero distance drives. I saw this when comparing the filtered Drives dashboard versus the same day Trip dashboard links off the Statistic page. This might be by design.

Right, it filters out also drives with duration_min=0 although there is distance>0. I proposed to show those also on the Drives dashboard in https://github.com/teslamate-org/teslamate/pull/3867

DrMichael avatar May 02 '24 07:05 DrMichael

One item I did find, testing a use-case from @towster - it seems the Drives dashboard filters out (either) zero duration or zero distance drives. I saw this when comparing the filtered Drives dashboard versus the same day Trip dashboard links off the Statistic page. This might be by design.

Right, it filters out also drives with duration_min=0 although there is distance>0. I proposed to show those also on the Drives dashboard in #3867

Found this PR about five minutes after making the comment above 🤦🏼 thanks for the find.

cwanja avatar May 02 '24 18:05 cwanja

@DrMichael Seems like there is an issue with this dashboard now:

teslamate_grafana | logger=provisioning.dashboard type=file name=teslamate t=2024-05-20T13:33:21.500815425Z level=error msg="failed to save dashboard" file=/dashboards/statistics.json error="could not resolve dashboards:uid:1EZnXszMl: Dashboard not found"

JakobLichterfeld avatar May 20 '24 13:05 JakobLichterfeld

Where does that come from? The build? Not sure, what that uid issue means...

(After a pull for edge I am still on 1.28.6-dev, might be an issue with my edge installation again... latest gives me 1.29.1. But no issue with Statistics...)

DrMichael avatar May 20 '24 15:05 DrMichael

@DrMichael Seems like there is an issue with this dashboard now:

Where does that come from?

Seems to be a dangling container issue on my end, grafana was on 1.29.1 and teslamate was on 1.28.5 hence the mismatch. Sorry for the inconvenience

JakobLichterfeld avatar May 20 '24 16:05 JakobLichterfeld

@DrMichael I am still seeing the timezone in 1.29.1 and it defaults to Berlin. Is anyone else seeing this? @JakobLichterfeld @jheredianet Screenshot 2024-05-29 at 8 51 35 PM

cwanja avatar May 30 '24 01:05 cwanja

Hmmm, looks good for me. You are running grafana:latest or grafana:edge?

I had a terrible time some weeks ago to get rid of old images. Perhaps try to purge all images of grafana and repull it.

DrMichael avatar May 30 '24 09:05 DrMichael

Not on edge. Running latest across all images. Just updated to 1.29.1 last night. Part of my upgrade script is a prune after pulling the latest image.

cwanja avatar May 30 '24 11:05 cwanja

@DrMichael I am still seeing the timezone in 1.29.1 and it defaults to Berlin. Is anyone else seeing this? @JakobLichterfeld @jheredianet

Can confirm Timezone setting on Statistics dashboard is still there with 1.29.1

JakobLichterfeld avatar May 30 '24 12:05 JakobLichterfeld

Hmmm, that is wierd. I removed all teslamate and grafana images, did a fesh pull. And have the new version...

DrMichael avatar May 30 '24 13:05 DrMichael

Is there something in the dashboard JSON I can verify to ensure it is in-fact the latest image?

cwanja avatar May 30 '24 13:05 cwanja

The new one uses the new $__timezone variable. The old one probably has Europe/Berlin in it.

DrMichael avatar May 30 '24 15:05 DrMichael

image Looks like it is still there. This at least from the live Grafana interface. Not sure how to dive into the code in the Docker container specifically (if necessary).

image image

cwanja avatar May 30 '24 16:05 cwanja

The screenshots are from github master? You can check in the image. It is in /dashboards.

DrMichael avatar May 31 '24 07:05 DrMichael