Feature Request: Project node count

[claudekenni]

I think it would be great if we could monitor the node count for certain projects.

This number can be queried by running a request against GET /api/14/project/[PROJECT]/resources and then counting the number of entries.

[phsmith]

Hi @claudekenni, thanks for using the rundeck_exporter and for your suggestion. I'll take a look at your request and get back to you soon.

[phsmith]

@claudekenni I was thinking about getting the nodes per project and showing them like this:

# HELP rundeck_project_node_info Rundeck project node info
# TYPE rundeck_project_node_info gauge
rundeck_project_node_info{description="Rundeck server node",hostname="4d6cf9461e16",instance_address="localhost:4440",nodename="4d6cf9461e16",osArch="amd64",osFamily="unix",osName="Linux",osVersion="6.7.8-arch1-1",project_name="test-1"} 1.0
rundeck_project_node_info{description="Rundeck server node",hostname="4d6cf9461e16",instance_address="localhost:4440",nodename="4d6cf9461e16",osArch="amd64",osFamily="unix",osName="Linux",osVersion="6.7.8-arch1-1",project_name="test-2"} 1.0

This approach gives you more options for setting up a query to display in your dashboard. What do you think?

[claudekenni]

@phsmith Is this broken down by individual nodes, so each node will have one line in the output? If so it might be a problem due to us having many nodes and I'm not sure how this would scale with thousands of nodes.

[phsmith]

Actually it will be broken down by nodes per project. But instead of that approach I can do as you suggested first to have a project node counter like:

rundeck_project_nodes_total{instance_address="localhost:4440",project_name="test-1"} 2.0

How does that sound?

[claudekenni]

That sounds great.

[phsmith]

All right @claudekenni, new release v2.6.5 deployed and available with the new rundeck_project_nodes_total metric.

[claudekenni]

Awesome thanks.

[claudekenni]

This is the extra ACLs I had to add to the Rule

---
by:
  username: exporter
description: Allow [read] for node
for:
  node:
  - allow:
    - read
    match:
      nodename: .*
context:
  project: .*

---
by:
  username: exporter
description: Allow [read] for (All) node
for:
  resource:
  - allow:
    - read
    equals:
      kind: node
context:
  project: .*

[phsmith]

Amazing! Thanks for sharing that. I'll update it in the README.md file.

[claudekenni]

@phsmith FYI It looks like this impacts the /var/log/rundeck folder, specifically the rundeck.audit.log file. For us with around 5k nodes, that file grows about 2GB per hour and the default logrotation was not enough to catch it and do the rotation so the filesystem ended up 100% full.

[phsmith]

Ouch, that's a lot.

Since you have audit logs enabled, it will certainly log all exporter requests. You may want to try the projects execution cache option (--rundeck.projects.executions.cache), but keep in mind that it may affect the monitoring results.

I could be wrong, but I don't think there is anything in Rundeck's configs to not log certain users, it would be nice to have something like that.

[claudekenni]

Looks like the default logging is set to info for the audit log. Setting it to warn cuts down these messages drastically.

[phsmith]

Wow, cool! Yeah, I completely forgot about the log levels, so that's an amazing discovery.

[phsmith]

Hi @claudekenni,

Just wanted to let you know that a new version of the export has been released, v2.7.0.

To get the rundeck_project_nodes_total in this version, you'll need to pass the --rundeck.projects.nodes.info parameter or the RUNDECK_PROJECTS_NODES_INFO=True environment variable. That's because it has been reported that this metrics collection causes high CPU load when you have many projects in Rundeck. This metrics collection function is now also cached, as it may not change often.

Let me know if you find any problems, otherwise it would be great if we could close this issue.

Thanks!