flink icon indicating copy to clipboard operation
flink copied to clipboard

[FLINK-23589][flink-avro] Support microsecond precision for timestamp

Open matriv opened this issue 3 years ago • 6 comments

What is the purpose of the change

Add support for microsecond precision for timestamp in Avro.

Brief change log

  • Add support for microsecond precision for timestamp in AvroRowDataDeserializationSchema.
  • Adjust tests to use different digits for hours:mins:seconds
  • Adjust tests to use complete timestamp and time when testing micros

Verifying this change

This change added tests and can be verified as follows:

  • AvroRowDataDeSerializationSchemaTest#testSpecificType
  • Adjusted all other tests to use full data/time when testing timestamp/time with micros

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

matriv avatar Apr 20 '22 15:04 matriv

Microseconds are not currently supported for TIME, opened: https://issues.apache.org/jira/browse/FLINK-27331 to track this as it's blocked by https://issues.apache.org/jira/browse/FLINK-17224

matriv avatar Apr 20 '22 15:04 matriv

CI report:

  • 6356a39386fe80008e3518031bfa2f402bd41dfb Azure: SUCCESS
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Apr 20 '22 15:04 flinkbot

@matriv is this PR blocked by https://issues.apache.org/jira/browse/FLINK-17224 and https://issues.apache.org/jira/browse/FLINK-27331? Without them, will the converters btw Flink RowData and Avro GenericRecord work with the microsecond precision after this fix?

stevenzwu avatar Sep 30 '22 17:09 stevenzwu

I think this PR is not blocked by FLINK-27331 or FLINK-17224, though FLINK-17224 blocks FLINK-27331. This PR is for TIMESTAMP data type and the two other JIRAs are for TIME.

liuml07 avatar Jan 04 '23 00:01 liuml07

Hey @matriv Any idea why this is blocked so long? :) I really need this TIMESTAMP type precision fix on AvroToRowDataConverters, can I help with something to unblock?

asantoz avatar Jul 21 '24 23:07 asantoz

Hey @matriv Any idea why this is blocked so long? :) I really need this TIMESTAMP type precision fix on AvroToRowDataConverters, can I help with something to unblock?

Not really, it was never reviewed :(

matriv avatar Jul 31 '24 17:07 matriv

This PR is being marked as stale since it has not had any activity in the last 180 days. If you would like to keep this PR alive, please leave a comment asking for a review. If the PR has merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out to the community, contact details can be found here: https://flink.apache.org/what-is-flink/community/

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 90 days, it will be automatically closed.

github-actions[bot] avatar Jan 27 '25 18:01 github-actions[bot]

This PR has been closed since it has not had any activity in 120 days. If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open the PR and ask for a review.

github-actions[bot] avatar Mar 31 '25 06:03 github-actions[bot]