spark-rapids icon indicating copy to clipboard operation
spark-rapids copied to clipboard
verified publisher icon NVIDIA

[FEA] Support time zones that are not UTC

Open revans2 opened this issue 3 years ago • 5 comments

Is your feature request related to a problem? Please describe. This is related to #6603 but a super set of it. I will list out the issues here to complete both sets of work, and then reference some of them in #6603 too.

The following are the tasks to do:

Phase I (P0) - Release 23.12

  • [x] https://github.com/NVIDIA/spark-rapids/issues/6831
  • [x] https://github.com/NVIDIA/spark-rapids/issues/9737
  • [x] https://github.com/NVIDIA/spark-rapids/issues/6832
  • [x] https://github.com/NVIDIA/spark-rapids/issues/9802
  • [x] https://github.com/NVIDIA/spark-rapids/issues/9805
  • [X] https://github.com/NVIDIA/spark-rapids/issues/9804
  • [x] https://github.com/NVIDIA/spark-rapids/issues/9829 (might move to Phase II)
  • [x] https://github.com/NVIDIA/spark-rapids/issues/9915 (might move to Phase II)

Phase II (P0) - Release 24.02

  • [x] https://github.com/NVIDIA/spark-rapids/issues/9606
  • [x] https://github.com/NVIDIA/spark-rapids/issues/9815
  • [x] https://github.com/NVIDIA/spark-rapids/issues/9605
  • [x] https://github.com/NVIDIA/spark-rapids/issues/6836
  • [x] https://github.com/NVIDIA/spark-rapids/issues/9927
  • [ ] https://github.com/NVIDIA/spark-rapids/issues/10079
  • [x] https://github.com/NVIDIA/spark-rapids/issues/9801
  • [x] https://github.com/NVIDIA/spark-rapids/issues/6834
  • [x] https://github.com/NVIDIA/spark-rapids/issues/9747
  • [x] https://github.com/NVIDIA/spark-rapids/issues/9881
  • [x] https://github.com/NVIDIA/spark-rapids/issues/9855
  • [ ] https://github.com/NVIDIA/spark-rapids/issues/9633
  • [ ] https://github.com/NVIDIA/spark-rapids/issues/6840
  • [ ] https://github.com/NVIDIA/spark-rapids/issues/9997 2 kernel: Parse string to local time; convert local time to UTC time.
  • [ ] https://github.com/NVIDIA/spark-rapids/issues/10034
  • [ ] https://github.com/NVIDIA/spark-rapids/issues/10035
  • [ ] https://github.com/NVIDIA/spark-rapids/issues/10083

Phase III/Uncategorized (P1)

  • [ ] https://github.com/NVIDIA/spark-rapids/issues/6835
  • [x] https://github.com/NVIDIA/spark-rapids/issues/10006
  • [ ] https://github.com/NVIDIA/spark-rapids/issues/6838
  • [ ] https://github.com/NVIDIA/spark-rapids/issues/6841
  • [x] https://github.com/NVIDIA/spark-rapids/issues/6842
  • [ ] https://github.com/NVIDIA/spark-rapids/issues/6843
  • [ ] https://github.com/NVIDIA/spark-rapids/issues/6844
  • [ ] https://github.com/NVIDIA/spark-rapids/issues/6845
  • [ ] https://github.com/NVIDIA/spark-rapids/issues/6846
  • [ ] https://github.com/NVIDIA/spark-rapids-jni/issues/1588
  • [x] https://github.com/NVIDIA/spark-rapids/issues/9916
  • [ ] https://github.com/NVIDIA/spark-rapids-jni/issues/1571

Uncategorized

  • [ ] https://github.com/NVIDIA/spark-rapids/issues/9653
  • [ ] https://github.com/NVIDIA/spark-rapids/issues/9912
  • [ ] https://github.com/NVIDIA/spark-rapids/issues/9913
  • [x] https://github.com/NVIDIA/spark-rapids/issues/9914 (might move to Phase II)

Test improvements

  • [ ] #10039
  • [ ] #10040
  • [x] #10041
  • [ ] #141

Benchmarking

  • [ ] https://github.com/NVIDIA/spark-rapids/issues/10142

In general we are going to try to have timestamps follow what Spark does and store them in UTC. Then convert the inputs to the desired timezone whenever we need to operate on it in a different time zone.

We should also go through other operators and see if there are things that we can/should support once we can support timezones.

revans2 avatar Oct 18 '22 16:10 revans2

Not sure if FromUTCTimestamp support PST should be a sub task. CC @viadea

nvliyuan avatar Jul 27 '23 07:07 nvliyuan

Follow on issue: https://github.com/NVIDIA/spark-rapids/issues/9570

res-life avatar Oct 30 '23 03:10 res-life

Adding in some issues for testing too

https://github.com/NVIDIA/spark-rapids/issues/9627

https://github.com/NVIDIA/spark-rapids/issues/9633

revans2 avatar Nov 03 '23 22:11 revans2

Test cases statistics:

code snapshot TZ passed skipped xfailed xpassed total
Before xfail all the impacted cases #9773 UTC 19361 2210 373 283 22227
After xfail all the impacted cases #9773 non-UTC 7589 2214 3719 8613 22135
Introduced fine grained checker from #9719 non-UTC 20173 1411 358 382 22324

res-life avatar Nov 28 '23 00:11 res-life

Not planing work on this for release 24.04 @sameerz can we move it to next release?

res-life avatar Mar 15 '24 01:03 res-life