flink icon indicating copy to clipboard operation
flink copied to clipboard

[FLINK-17224][table] Support precision for TIME

Open snuyanzin opened this issue 2 years ago • 4 comments

What is the purpose of the change

The PR makes support of TIME precision (0-3 - currently only this range, I'm going to submit support for micros and nanos in the next PR under probably https://issues.apache.org/jira/browse/FLINK-17525). It includes casts, time related functions (CEIL, FLOOR, OVERLAPS, agg MAX, MIN, LEAD, LAG, singlevalue ) Also added support of TIME precision in csv, json, avro, some tests in Kafka (they use agg MAX e.g.)

Verifying this change

A number of tests in table-planner, avro, csv, json, kafka dealing with TIME

especially flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java and flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java checking different variations of casts to and from TIME

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): ( no)
  • The serializers: ( no)
  • The runtime per-record code paths (performance sensitive): ( no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no)
  • The S3 file system connector: ( no)

Documentation

  • Does this pull request introduce a new feature? ( no)
  • If yes, how is the feature documented? (not applicable )

snuyanzin avatar Jun 13 '23 19:06 snuyanzin

CI report:

  • 6ac1e6e41ae85c8dc049cc46dd6a19a6f72a6fb3 Azure: SUCCESS
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Jun 13 '23 20:06 flinkbot

@dawidwys since you are the reporter of the issue could you please have a look? // cc @wuchong since you are the reporter of a very similar https://issues.apache.org/jira/browse/FLINK-17525

snuyanzin avatar Jun 14 '23 05:06 snuyanzin

Rebased to solve the conflicts

snuyanzin avatar Jun 17 '24 09:06 snuyanzin

Thanks for your feedback

Could we change sth so that we don't lose data when a precision >3 is used? Could we throw an exception somewhere explicitly?

This seems a questionable thing how Flink should behave.

I checked it for Postgres where for TIME the precision >6 is not supported and Postgres continues working with loosing data (it puts warning in it's output log about that)

On the other side MySQL throws an exception.

I checked Flink SQL for

SELECT CAST('1111-11-11 11:11:11.123456789123' AS timestamp(12))

and it fails as

Caused by: java.time.format.DateTimeParseException: Text '1111-11-11 11:11:11.123456789123' could not be parsed, unparsed text found at index 29
	at java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2049)
	at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1874)
	at org.apache.flink.table.utils.DateTimeUtils.parseTimestampData(DateTimeUtils.java:413)
	at org.apache.flink.table.data.binary.BinaryStringDataUtil.toTimestamp(BinaryStringDataUtil.java:612)

I think we could add similar behavior for TIME

snuyanzin avatar Jun 17 '24 09:06 snuyanzin