paimon icon indicating copy to clipboard operation
paimon copied to clipboard
verified publisher icon apache

[metrics] Add io-metrics for paimon in Flink

Open zhuanshenbsj1 opened this issue 8 months ago • 6 comments

Purpose

  1. Monitor the input/output (IO) of Flink during read and write operations.

Tests

API and Format

Documentation

zhuanshenbsj1 avatar Apr 28 '25 10:04 zhuanshenbsj1

1、You add lots of metric, I suggest you sent the discuss email to [email protected] first. 2、modify the doc.

wwj6591812 avatar Apr 29 '25 10:04 wwj6591812

1、You add lots of metric, I suggest you sent the discuss email to [email protected] first. 2、modify the doc.

Roger that.

zhuanshenbsj1 avatar Apr 30 '25 01:04 zhuanshenbsj1

I'v done refactor for making with MetricRegistry public in https://github.com/apache/paimon/pull/5578

JingsongLi avatar May 08 '25 05:05 JingsongLi

I'v done refactor for making with MetricRegistry public in #5578

rebase on this && add uts & all checks passed, cc~ @JingsongLi

zhuanshenbsj1 avatar May 11 '25 12:05 zhuanshenbsj1

Please only use TableWrite.withMetricRegistry and TableRead.withMetricRegistry, do not introduce any API for metrics.

And please make sure only io metric enabled, the file io can be MetricsFileIO.

If we only use TableRead.withMetricRegistry, it would require too may modifications. However, in reality, we're always using and modifying the same AbstractFileStoreTable fileio reference. I've attempted to register metrics directly to the AbstractFileStoreTable within the ReadBuilder. This way, all subsequent operations like new scan, new streamscan, or new read can directly reuse the metrics collected by the AbstractFileStoreTable.

zhuanshenbsj1 avatar May 14 '25 09:05 zhuanshenbsj1

@JingsongLi Could you please take a look at this PR again for any remaining issues， thank ~

zhuanshenbsj1 avatar May 21 '25 07:05 zhuanshenbsj1