flytekit icon indicating copy to clipboard operation
flytekit copied to clipboard

add timestamp transformer

Open dansola opened this issue 3 months ago • 0 comments

Why are the changes needed?

What changes were proposed in this pull request?

This workflow

@fl.task(container_image=image)
def get_timestamp() -> pd.Timestamp:
    return pd.Timestamp("2024-09-29 15:30:45")


@fl.workflow
def timestamp_wf() -> pd.Timestamp:
    return get_timestamp()

gives this error:

TypeTransformerFailedError: Failed to convert 
outputs of task 'timestamp_wf.get_timestamp' at 
position 0.
Failed to convert type <class 
'pandas._libs.tslibs.timestamps.Timestamp'> to type 
<class 'pandas._libs.tslibs.timestamps.Timestamp'>.
Error Message: Expected value of type <class 
'datetime.datetime'> but got '2024-09-29 15:30:45' 
of type <class 
'pandas._libs.tslibs.timestamps.Timestamp'>.

We need a pandas timestamp type transformer.

How was this patch tested?

Tested locally and remotely. For remote run, just installed flytekit from the da72500c93e1ff1107a24f1e67eb453a443e4d82 commit on the image.

Check all the applicable boxes

  • [ ] I updated the documentation accordingly.
  • [ ] All new and existing tests passed.
  • [ ] All commits are signed-off.

Related PRs

Docs link

Summary by Bito

This pull request introduces the PandasTimestampTransformer class to convert between pandas.Timestamp and Flyte DATETIME literals, addressing a type conversion error in workflows. It includes updates to import statements and has been tested for functionality both locally and remotely.

dansola avatar Sep 29 '25 19:09 dansola