spark-datetime
spark-datetime copied to clipboard
Bucket function fails for longer time periods > 1 month
trafficstars
Joda Time, periods cannot deal with starts and ends that have a difference > 1 month see:http://stackoverflow.com/questions/8793475/joda-time-all-in-minutes
This line therefore:
https://github.com/SparklineData/spark-datetime/blob/master/src/main/scala/org/sparklinedata/spark/dateTime/Functions.scala#L400
results in
Caused by: java.lang.UnsupportedOperationException: Cannot convert to Duration as this period contains months and months vary in length
at org.joda.time.Period.checkYearsAndMonths(Period.java:1567)
at org.joda.time.Period.toStandardDuration(Period.java:1549)
at org.sparklinedata.spark.dateTime.FunctionsExt$timeBucketFn$.apply(FunctionsExt.scala:400)
perhaps this needs to be changed to use something like:
val d0: Duration = p.toStandardDuration
assert((dt.getMillis>origin.getMillis))
FieldUtils.safeDivide((dt.getMillis-origin.getMillis), d0.getMillis)