scylla-cluster-tests icon indicating copy to clipboard operation
scylla-cluster-tests copied to clipboard

feature(monitoring): set monitoring branch to 4.7

Open amnonh opened this issue 1 year ago • 9 comments

  • [x ] I didn't leave commented-out/debugging code

amnonh avatar Mar 24 '24 09:03 amnonh

I've mark 2 provision test to run

we'll check their monitoring output, and if o.k., we can merged this one

fruch avatar Mar 25 '24 08:03 fruch

@amnonh

why we don't have images for this branch ? (i.e. in other PR i'm trying out the images, and I only see 4.6.2)

fruch avatar Mar 28 '24 14:03 fruch

our dashboard becomes a bit more broken with this version: image

for some reason the default view show only the metric name, and not the values (we'll need to fix that)

fruch avatar Mar 28 '24 15:03 fruch

I don’t want to merge it if it breaks our view

roydahan avatar Mar 28 '24 15:03 roydahan

os monitoring also doesn't work properly. I could see only one instance metrics. Tested on branch-perf-v14: image

soyacz avatar Apr 04 '24 11:04 soyacz

The image @soyacz sent suggested there could be an issue with the IPv6 configuration. It's weird to see the port number in the dashboard (with the panels that show no data). I would suggest checking the Prometheus targets configuration

amnonh avatar Apr 04 '24 12:04 amnonh

@fruch please correct me if I'm wrong, but now that you moved to monitor images, it's not enough to just change the monitor branch.

roydahan avatar Apr 05 '24 02:04 roydahan

@fruch please correct me if I'm wrong, but now that you moved to monitor images, it's not enough to just change the monitor branch.

We haven't yet merged it, but yes it changes a bit this flow of updating, hence why I've asked here where are the images

fruch avatar Apr 05 '24 03:04 fruch

@amnonh possibly problem I'm seeing is due missing backport of https://github.com/scylladb/scylla-cluster-tests/pull/6975 to branch-perf-v14

soyacz avatar Apr 05 '24 12:04 soyacz

our dashboard becomes a bit more broken with this version: image

for some reason the default view show only the metric name, and not the values (we'll need to fix that)

@amnonh can you help us figure this one ? its what holding us from merging this (and the fact we now need to specify also the images for 4.7)

fruch avatar May 28 '24 05:05 fruch

@yaronkaikov

can you clarify where the monitoring images should be found ? on which account ? and why aren't they public ?

fruch avatar Jun 04 '24 09:06 fruch

I've check the latest runs of this PR, and the issue with the nemesis panel isn't there anymore:

image

once I'll find the right image for GCE, this is good to go

fruch avatar Jun 09 '24 08:06 fruch