flink icon indicating copy to clipboard operation
flink copied to clipboard

[FLINK-33697][state][metrics] Trace RocksDBIncremental remote files download time

Open pnowojski opened this issue 1 year ago • 4 comments

This PR builds on top of https://github.com/apache/flink/pull/23908

Brief change log

Please check individual commit messages

Verifying this change

This change has been manually tested and added a unit test that metric has been created.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

pnowojski avatar Jan 03 '24 15:01 pnowojski

CI report:

  • d856fd61fa7548d4f8de404ec368e2b7ec4e9cdd Azure: SUCCESS
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Jan 03 '24 15:01 flinkbot

Thanks for the review!

Why only add the download metric? While you are at this, you could have also simply added metrics for, e.g., merge/clip.

That was the scope of the FLIP. Partially because I wanted to keep the scope as small as possible to minimise required efforts. We can always add more metrics in the future.

pnowojski avatar Jan 04 '24 14:01 pnowojski

@flinkbot run azure

pnowojski avatar Jan 09 '24 13:01 pnowojski

I think that's a fair approach. I agree very few users implement state backends and if they do they are knowledgeable enough to easily migrate their code. Moreover we changed this interface rather freely in the past adding and removing arguments to its methods. Therefore +1 for the approach.

dawidwys avatar Jan 11 '24 09:01 dawidwys