client-go icon indicating copy to clipboard operation
client-go copied to clipboard

metric: Add memdb size and key count metric

Open longfangsong opened this issue 3 years ago • 6 comments

Add size and keys count metric for memdb

longfangsong avatar May 13 '22 02:05 longfangsong

There could be multiple MemDBs. ~I think Gauge for MemdbSize is inappropriate. Maybe a Histogram?~ Gauge makes sense. We just need to differentiate memdbs.

ekexium avatar May 19 '22 08:05 ekexium

~I'm wondering about the purpose of these metrics. It answers "how much data is in memdb" but cannot tell us a lot about "how much memory is used by memdb".~

I see, it's for measuring the actual memory consumed by memdb. db.size is just the sum of lengths of keys and values we put in which is the theoretical lower bound of the actual memory consumption. Could we consider calculating the memories allocated for the memdbArenas?

We can run some experiments with large transactions and compare our metrics with the profiling results to see if it's accurate.

ekexium avatar May 19 '22 09:05 ekexium

We don't have a clear idea of the relationship between db.size and the memory allocated. I'm thinking could we export both of them so that we will be able to estimate the memory usage of certain kinds of jobs, e.g.insert into select a large table?

ekexium avatar May 20 '22 04:05 ekexium

#520 tracks the memory allocated to the arena. That could be reused here.

ekexium avatar Jun 16 '22 03:06 ekexium

@longfangsong Please address the above comments.

cfzjywxk avatar Jun 28 '22 08:06 cfzjywxk

@longfangsong

Please address the above comments.

I plan to continue this pr after #520 is done.

longfangsong avatar Jun 28 '22 09:06 longfangsong

@cfzjywxk @MyonKeminta do we want to continue this?

disksing avatar Aug 29 '22 03:08 disksing

closing due to inactivity. Please reopen it if need.

disksing avatar Sep 13 '22 05:09 disksing