clickhouse.rs icon indicating copy to clipboard operation
clickhouse.rs copied to clipboard

Expose Inserter metrics / debug info for performance monitoring

Open tomatopunk opened this issue 2 months ago • 3 comments

Use case

Now, clickhouse.Inserter like black box - it's hard to see what's happening with performance

I want to be able to:

  • monitor how many rows Inserter actually has buffered at any moment.
  • when and why a flush is triggered (manual commit(), with_period(), with_max_rows(), etc.).
  • see the size of each batch of data actually sent to ClickHouse.
  • measure the local-to-server round-trip time per flush (commit RTT).
  • track total rows sent and batches committed over time.

Describe the solution you'd like

expose client-side flush/debug metrics in prometheus format like:

  • buffered_rows / buffered_bytes
  • total_rows_sent / total_batches
  • last_batch_rows / last_batch_bytes
  • last_batch_rtt_seconds
  • last_flush_reason

Describe the alternatives you've considered

add inserter function:

  • buffered_rows() → current buffered row count
  • buffered_bytes() → estimated serialized size in bytes
  • on_flush(callback: Fn(&FlushInfo)) → hook fired before/after every flush
  • with_debug_logging(enabled: bool) → enable structured debug logs for Inserter

Additional context

tomatopunk avatar Nov 11 '25 06:11 tomatopunk

We're absolutely interested in adding observability across the client. The biggest question is how much we want to standardize across our various language clients (e.g. the logs and spans we want to emit), so we need to take some time to go through the different feature requests for metrics across repos and aggregate all the different information people are wanting to see.

abonander avatar Nov 11 '25 17:11 abonander

See also #323

abonander avatar Nov 11 '25 17:11 abonander

I fully agree that it takes some time to standardize observability across all language clients, but in the meantime, could we provide some hooks and fields — like inserter.pending() — so that we can design our own metrics as a substitute? At the moment, the information exposed is really too limited.

tomatopunk avatar Nov 12 '25 02:11 tomatopunk