performancecopilot
Multiple vector hosts cannot be configured properly from higher version Grafana
Hi,
Data source: Vector
Describe the bug I followed the grafana-pcp quickstart guide and was able to query metrics properly from PCP Vector data source with a predefined target host. Currently I am trying to configure my grafana-pcp dashboard with multiple hosts following the guide here: https://grafana-pcp.readthedocs.io/en/latest/guides/multiple-vector-hosts.html
However I am seeing some errors while following the guide, below are the steps I tried with screenshots. Would you please kindly help to identify or clarify if I am making mistakes? Or provide any hints to configure it properly if maybe the doc is a bit outdated? Thanks!
To Reproduce
-
Follow the "Setup the Vector data source" section As the doc shows, the error is expected.
-
Create a new dashboard with variables In this step I created a new dashboard with a textbox variable "host"
-
Configure the dashboard The problem is in this step: Without the URL set in the data source, it seems that Grafana won't allow me to enter the PCP metric name and the URL field. Any hints on this one? Thanks!
Versions (please complete the following information):
- Performance Co-Pilot: 5.3.6
- Grafana: v11.0.0
- grafana-pcp: 5.0.0
- Redis (if applicable): N/A
Hi @natoscott @sfeifer @andreasgerstmayr , would you please kindly provide any hint on this issue? Thanks very much!
One update: Using Grafana 9.0.0, I am able to switch the host by changing the URL variable from the default "PCP Vector Checklist" dashboard imported from the grafana-pcp plugin, and the dashboard works well. However using same configuration/setup, if I switch back to Grafana 11.0.0 I will get a "Network Error: NetworkError when attempting to fetch resource", the dashboard will be empty with the error in this case.
Any idea why or where I can start to debug? Thanks!
Is there any update on this issue? I just set up Grafana and Cockpit PCP on a few systems and would like them to be in the same data source. I can work around it for now by adding multiple data sources, but that's not ideal.
Hi @FIrgolitsch. There has not been any update on this so far. Can you confirm the versions of grafana, pcp, and grafana-pcp you are using? I understand the original issue as specific to certain versions and am wondering if the issue you are experiencing has some differences from this one (in which case we would open a new issue).
I'm using Grafana 12.2.0, PCP 6.2.0 and Grafana-PCP 5.3.0. Interestingly, I also noticed this just now in Grafana's plugin administration when checking the version:
Nothing major, just though you might want to know.
I initally followed this setup from Cockpit: https://cockpit-project.org/blog/pcp-grafana.html It's 4 years old at this point, but works fine for the most part. The data source has changed from Redis to Valkey (though Cockpit-PCP still uses Redis, or maybe pmproxy, I'm not too sure on the details of Cockpit-PCP) but functionality is the same. The only thing is that I can't get mutiple hosts in the same data source, with multiple data sources working fine. I have not tired with versions other than this, as it was a brand new setup and I used all the lastest versions that were available.
@FIrgolitsch Thanks for that info and pointing us towards the "Latest Version" conflict. It sounds like the issue you are experiencing is different from the original one, though. Can you open a new issue with the relevant information where we can continue this discussion?
The original issue here has to do with modifying the url for the pcp vector datasource in a textbox in the dashboard (which allows for quickly switching between different hosts in a pcp vector dashboard). I understand your issue as a problem with viewing multiple hosts from the same pcp valkey datasource (correct me if I'm wrong here).
One thought I have is that the database and/or pmloggers are not setup as intended. What you want is one singular valkey database with the metrics from all the hosts you want to look at. This database will then be used for the pcp valkey datasource and allows polling metrics from the different hosts. What could be happening right now is each host having its own valkey database with just the metrics for that host. This would result in being able to create multiple different datasources from the unique valkey databases that visualize the different hosts, but none that allow visualizing the metrics from all the hosts.
No, my issue is the same. When I enter different hosts in the box it does not connect to them. My workaround is having multiple data sources. What you are saying is correct, though. Because the logging is managed by Cockpit via Cockpit-PCP, each has a different (I think) Redis database for the logging. So when I add the multiple data sources tot Grafana, it's one data source per host leading to multiple Valkey databses for Grafana.
I have no experience with Valkey. If you have some resources where I can find out more about setting up, I would be grateful. With what you said my thought is now to have a seperate Valkey database which ingests data from all my Cockpit-PCP instances. In turn, Grafana reads that datasource so it will have all the hosts in a singluar source. Is this the right way of thinking?
I understand the issue looks identical for the user, but on the backend the pcp valkey and pcp vector datasources work differently. Without getting too nitty gritty, changing the "host" for the pcp vector datasource directly modifies the datasource itself (i.e. changes the url for the datasource). For the pcp valkey datasource, changing the "host" does not change the url for the datasource. Instead, it adds an argument to every query to collect the metrics for just the specified host. Does that make sense and make clear why the initial issue is different from this one?
Here's some documentation on setting up pmlogger so you can log metrics from a system you want to monitor on a central host. After setting up pmlogger, pmproxy will automatically start populating the database (valkey) on the collector system. No additional setup should be needed other than potentially needing to restart the services.
Here's a visual representation of the setup I think you are looking for. The "key-server" in the image is the database that stores the metrics for the monitored systems.
Yes, that fully clears it up! Thanks a lot for the resources, which'll go a long way.