fromdate and todate changes time by one hour on macOS in summer

[simon-biber]

trafficstars

I'm running jq version 1.6 on macOS 10.15 and trying to use jq to convert a starting date/time plus a given number of milliseconds (in the .timestamp of my JSON input) into the corresponding date/time. I noticed that all the outputs are 1 hour later than they should be. This shouldn't be happening as both input and output are in UTC time. Something internal to jq seems to be influenced by the daylight-savings rules of my local time zone.

My code looks like: (.timestamp/1000.0 + ("2019-10-19T09:02:29Z" | fromdate) | todate)

Experimenting with this, it happens whenever the given date is in summer in my local time zone (ACDT UTC+10:30). For example, if input is midnight on 1-Dec-2019 the output is 1:00am. Note that December is summer in Australia.

$ echo '"2019-12-01T00:00:00Z"' | jq 'fromdate | todate'
"2019-12-01T01:00:00Z"

[jmai444]

I can corroborate this on my T470 running Linux and jq version 1.6.

From a long list of UTC timestamps, those falling during DST of my local timezone (also Australia) are ahead by one hour when converted to milliseconds using fromdate.

[yuv422]

I believe this has already been fixed in master. It was fixed by this commit.

https://github.com/stedolan/jq/commit/3c5b1419a278dfb192666b33197dc182c670290d

[SpicyLemon]

I can confirm that it is fixed in master (but not 1.6). With version 1.6:

> jq --null-input ' { "now_good": now | trunc, "now_bad": now | todate | fromdate } | .diff = .now_bad - .now_good '
{
  "now_good": 1584586842,
  "now_bad": 1584590442,
  "diff": 3600
}

On master:

> ./jq --null-input ' { "now_good": now | trunc, "now_bad": now | todate | fromdate } | .diff = .now_bad - .now_good '
{
  "now_good": 1584586851,
  "now_bad": 1584586851,
  "diff": 0
}

[mtheck]

Really annoying bug, wonder why it takes so much time to have a new release of jq?!

[sirmax]

I'm currently using this kludge in my scripts. Should work correctly on both 1.6 and next versions.

# workaround for https://github.com/stedolan/jq/issues/2001
def fromdate1: (. | fromdate) as $t1 | ($t1 | todate | fromdate) as $t2 | $t1 - ($t2 - $t1);

[fgzupper]

Word of caution -- that kludge doesn't cover the edge of DST. (2021-11-07T02:00:00Z is the next time we drop off of DST)

echo '"2021-11-07T00:00:00Z"' |\
 jq 'def fromdate1: (. | fromdate) as $t1 | ($t1 | todate | fromdate) as $t2 | $t1 - ($t2 - $t1);
  . | fromdate1 | todate'

"2021-11-07T01:00:00Z"

Reason being:

> echo '"2021-11-07T00:00:00Z"' |  jq '. | fromdate | todate'
"2021-11-07T01:00:00Z"
> echo '"2021-11-07T01:00:00Z"' |  jq '. | fromdate | todate'
"2021-11-07T01:00:00Z"

So the difference between $t2 and $t1 is zero in that one case, yet $t1 is still an hour off from what it should be.

[jfagoagas]

If anyone is suffering this, check the following code. https://github.com/stedolan/jq/blob/77417c1335a12c4ceef469caf38c0cbfb6315b45/src/builtin.c#L1266-L1280

TL;DR mktime() has side-effects and anyways, returns time in the local timezone, not UTC. jq documentation at Date says jq provides some basic date handling functionality, with some high-level and low-level builtins. In all cases these builtins deal exclusively with time in UTC.

So, if your timezone is different from UTC, even using fromdateiso8601 the date returned is in your local timezone and not in UTC.

[jfagoagas]

A possible fix is to set TZ=UTC environment variable after calling jq.

➜  ~ export TZ=UTC; echo '"2022-06-14T09:16:11+00:00"' |  jq '. | sub("\\+00:00";"Z") | strptime("%Y-%m-%dT%H:%M:%SZ")| mktime | todate'
"2022-06-14T09:16:11Z"

[kbolino]

You can also simply set the timezone inline when you run the command, e.g.

$ echo '"2022-08-29T19:00:00Z"' | TZ=US/Eastern jq fromdateiso8601
1661803200
$ date --date=@1661803200 -u -Iseconds
2022-08-29T20:00:00+00:00

vs.

$ echo '"2022-08-29T19:00:00Z"' | TZ=UTC jq fromdateiso8601
1661799600
$ date --date=@1661799600 -u -Iseconds
2022-08-29T19:00:00+00:00

Instead of exporting the variable, which might adversely affect later commands, this way constrains the override to just the invocation of jq.

[thaliaarchi]

@emanuele6 I can confirm, that this works on macOS on 1.7.

$ jq-1.7 -n 'now | (todate | fromdate) - trunc'
0
$ jq-1.6 -n 'now | (todate | fromdate) - trunc'
3600

[emanuele6]

@thaliaarchi Thank you!