perspective icon indicating copy to clipboard operation
perspective copied to clipboard

Add `date()`/`datetime()` functions to ExprTK suitable for simple data cleaning.

Open texodus opened this issue 3 years ago • 0 comments

Feature Request

Description of Problem:

Perspective exclusively represents datetime types as POSIX milliseconds values internally, and localizes them to the perspective-viewer's native time zone on display. However, it has some complex rules on how it regards naive datetime types on various sytems, and its not always the case that mis-encoded data sets can be easily normalized before being loaded into a <perspective-viewer>, so it would be nice to be able to use ExprTK to fix datetime columns that are either mis-inferred or flat-out wrong.

Note it would also be great to just document how datetime handling currently works all-around, as the existing docs are buried in the Python details.

Potential Solutions:

A variety of ExprTK functions could be added to make simple correction easy, though this is not an exhaustive list and the specifics of naming/priority are up for debate.

  • to_posix()/from_posix() to convert datetime columns explicitly to POSIX float values and back. Note this already basically works with casting except that the type-checked frequently barfs (or allows invalid) types, so this may be as easy as getting proper casting/checking to work here?
  • hours()/minutes()/etc time delta creation functions which return POSIX float types, so we can do e.g. datetime(float("date col") + hours(1)) to manually offset timezones.
  • apply_dst() to offset only datetime values that fall within DST for the locale.
  • Add support for numeric parameters to bucket(), e.g. bucket("time col", '15s'), so that arbitrary buckets can generated without multiple bucket steps.
  • Add conversion functions between date and datetime that work correctly. Currently, date casts work in that they cast to their integer representation e.g. the integer 20220628 for Jun 28, 2022.

texodus avatar Jun 29 '22 23:06 texodus