jq
jq copied to clipboard
jqlang
fromdate yields incorrect result
Describe the bug Tried to parse dates to calculate relative time between an ISO date and "now", got negative results which shouldn't be possible.
To Reproduce
$ date -ud @$(jq -n '"2021-04-22T15:55:48Z" | fromdate')
Thu 22 Apr 2021 04:55:48 PM UTC
As you can see, the parsed date is off by one hour.
Expected behavior I expect the time to match the input.
Environment (please complete the following information):
- OS and Version: Linux Debian Testing
- jq version: 1.6
Additional context The problem does not occur on all machines. I suspect it has to do with the configuration of the system's Real Time Clock. For compatibility with my dual-booted machine's Windows install I have
Local time: Thu 2021-04-22 18:16:12 CEST
Universal time: Thu 2021-04-22 16:16:12 UTC
RTC time: Thu 2021-04-22 17:21:45
Time zone: Europe/Amsterdam (CEST, +0200)
System clock synchronized: yes
NTP service: active
RTC in local TZ: yes
Most significantly: RTC in local TZ: yes. It feels like, jq is parsing the date as if it were my local timezone, although that would explain a 2-hour offset, not a 1-hour offset... So maybe there is some compensation in the wrong direction?
I just realized the RTC time is off by one, but linux is reporting correct time, that would explain a 1-hour offset. I'm guessing because I haven't booted into windows since the DST switch. Regardless, I think this still qualifies as a bug.
Are you sure it's an RTC issue? On a single-boot mac, I'm seeing this for any timezone in daylight savings time.
$ echo $'now | floor\t| todate \t| fromdate\toffset\tTZ'; \
> for tz in EST PST America/Montevideo America/Phoenix Australia/Sydney Z EST5EDT PST9PDT America/New_York America/Los_Angeles; do \
> echo $tz | TZ=$tz jq -R -r '[(now|floor), (now|todate), (now|todate|fromdate), (now|todate|fromdate)-(now|floor), .] | @tsv'; \
> done
now | floor | todate | fromdate offset TZ
1619735359 2021-04-29T22:29:19Z 1619735359 0 EST
1619735359 2021-04-29T22:29:19Z 1619735359 0 PST
1619735359 2021-04-29T22:29:19Z 1619735359 0 America/Montevideo
1619735359 2021-04-29T22:29:19Z 1619735359 0 America/Phoenix
1619735359 2021-04-29T22:29:19Z 1619735359 0 Australia/Sydney
1619735359 2021-04-29T22:29:19Z 1619735359 0 Z
1619735359 2021-04-29T22:29:19Z 1619738959 3600 EST5EDT
1619735359 2021-04-29T22:29:19Z 1619738959 3600 PST9PDT
1619735359 2021-04-29T22:29:19Z 1619738959 3600 America/New_York
1619735359 2021-04-29T22:29:19Z 1619738959 3600 America/Los_Angeles
¢ jq -V
jq-1.6
Same result from a Docker image running on EC2:
now | floor | todate | fromdate offset TZ
1619737232 2021-04-29T23:00:32Z 1619737232 0 EST
1619737232 2021-04-29T23:00:32Z 1619737232 0 PST
1619737232 2021-04-29T23:00:32Z 1619737232 0 America/Montevideo
1619737232 2021-04-29T23:00:32Z 1619737232 0 America/Phoenix
1619737232 2021-04-29T23:00:32Z 1619737232 0 Australia/Sydney
1619737232 2021-04-29T23:00:32Z 1619737232 0 Z
1619737232 2021-04-29T23:00:32Z 1619740832 3600 EST5EDT
1619737232 2021-04-29T23:00:32Z 1619740832 3600 PST9PDT
1619737232 2021-04-29T23:00:32Z 1619740832 3600 America/New_York
1619737232 2021-04-29T23:00:32Z 1619740832 3600 America/Los_Angeles
/app # jq -V
jq-master-v3.7.0-4757-gc31a4d0fd5
Here's a workaround function that can go in the .jq file:
# Workaround for https://github.com/stedolan/jq/issues/2304
def fromdatefixed:
fromdate - (now|floor|((todate|fromdate)-.))
;
Example:
$ jq -n '"2021-05-12T13:14:15Z" | [., (fromdate|todate), (fromdatefixed|todate)]'
[
"2021-05-12T13:14:15Z",
"2021-05-12T14:14:15Z",
"2021-05-12T13:14:15Z"
]
I'm also seeing this on a Mac (macOS 11.3.1, jq version 1.6, local time set to Pacific Daylight Time). A small repro case showing that fromdateiso8601 / todateiso8601 don't round-trip correctly (an extra hour gets added):
$ echo '"2021-06-02T05:00:00Z"' | jq -r 'fromdateiso8601 | todateiso8601'
2021-06-02T06:00:00Z
Seeing this as well on a Mac (10.15.7), jq version of 1.6, local time is EDT.
╭─pchernikov@pchernikov-mbp ~
╰─➤ echo '"2022-05-26T02:15:35Z"' | jq ' . | fromdateiso8601' 4 ↵
1653534935
╭─pchernikov@pchernikov-mbp ~
╰─➤ date -d "2022-05-26T02:15:35Z" +"%s"
1653531335
I too am seeing a discrepancy between mac and linux.
On Ubuntu 22.04 with jq 1.6
This one works fine.
$ TZ=Z date -d '2022-03-13T02:00:00Z' '+{"seconds": %s, "timestamp": "%FT%TZ"}' | jq '[.seconds, (.timestamp | fromdateiso8601)]'
[
1647136800,
1647136800
]
On MacOS 12.4 with jq 1.6
This one has a discrepancy.
$ TZ=Z gdate -d '2022-03-13T02:00:00Z' '+{"seconds": %s, "timestamp": "%FT%TZ"}' | jq '[.seconds, (.timestamp | fromdateiso8601)]'
[
1647136800,
1647140400
]
Specific date boundary
However, if I subtract 1 second from the time I used in the tests above, there is no problem:
On MacOS 12.4 with jq 1.6 before 2022-03-13T02:00:00Z
$ TZ=Z gdate -d '2022-03-13T01:59:59Z' '+{"seconds": %s, "timestamp": "%FT%TZ"}' | jq '[.seconds, (.timestamp | fromdateiso8601)]'
[
1647136799,
1647136799
]
This is a different behavior from what was originally reported though, because that report was also before 2022-03-13T02:00:00Z but it did exhibit an inconsistency.
Not date related
To rule out the possibility that date is part of the problem, here is the same data from echo on macOS:
$ echo '
[
{
"seconds": 1647136799,
"timestamp": "2022-03-13T01:59:59Z"
},
{
"seconds": 1647136800,
"timestamp": "2022-03-13T02:00:00Z"
}
]' | jq '.[] | [.seconds, (.timestamp | fromdateiso8601)]'
[
1647136799,
1647136799
]
[
1647136800,
1647140400
]
I suspect this might be Daylight-Savings related?
For the current datetime, Brisbane and Melbourne (timezones currently overlapping to produce the same local time):
$ echo '"2022-07-18T02:26:01Z"' | TZ=Australia/Brisbane jq '. | fromdateiso8601'
1658111161
$ echo '"2022-07-18T02:26:01Z"' | TZ=Australia/Melbourne jq '. | fromdateiso8601'
1658111161
Then in the future (October), when Melbourne goes into DST:
$ echo '"2022-10-02T02:26:01Z"' | TZ=Australia/Melbourne jq '. | fromdateiso8601'
1664681161
$ echo '"2022-10-02T02:26:01Z"' | TZ=Australia/Brisbane jq '. | fromdateiso8601'
1664677561
At the DST Boundary[^1] (this is incorrect, but it describes what is attempting to happen):
$ echo '"2022-10-02T01:59:59Z"' | TZ=Australia/Melbourne jq '. | fromdateiso8601'
1664675999
$ echo '"2022-10-02T01:59:59Z"' | TZ=Australia/Brisbane jq '. | fromdateiso8601'
1664675999
$ echo '"2022-10-02T02:00:00Z"' | TZ=Australia/Melbourne jq '. | fromdateiso8601'
1664679600
$ echo '"2022-10-02T02:00:00Z"' | TZ=Australia/Brisbane jq '. | fromdateiso8601'
1664676000
[^1]: At 2am on the 2nd of October, in Melbourne's local time, is when the DST transition occurs. The date being parsed is October 2nd 2am in UTC, which is about 10 hours after the transition is actually meant to occur.
Now that I've looked further around in the issues list, is this a duplicate of https://github.com/stedolan/jq/issues/2001 ?
FWIW there is a pure go implementation of jq that does not have this bug:
https://github.com/itchyny/gojq
$ TZ=Z gdate -d '2022-03-13T02:00:00Z' '+{"seconds": %s, "timestamp": "%FT%TZ"}' | gojq '[.seconds, (.timestamp | fromdateiso8601)]'
2022-08-04T10:15:56-0400
[
1647136800,
1647136800
]
It's possible that just releasing a new version of jq with new libraries would solve this. jq 1.6 is coming up on 4 years old.
This can probably be closed because jq 1.7.1 fixes the problem:
$ sw_vers
ProductName: macOS
ProductVersion: 14.2.1
BuildVersion: 23C71
$ jq --version
jq-1.7.1
$ TZ=Z gdate -d '2022-03-13T02:00:00Z' '+{"seconds": %s, "timestamp": "%FT%TZ"}' | gojq '[.seconds, (.timestamp | fromdateiso8601)]'
[
1647136800,
1647136800
]
