Add a StorageGRID collector

[rvalab]

As a storage administrator, I'm looking to have a single tool to monitor my NetApp environment. ActiveIQ Unified Manager, Harvest and NABox are greats tools but focused only to ONTAP. It will be great if we can have StorageGRID monitoring inside Harvest.

[cgrinds]

thanks for the feature request @rvalab - this is on the backlog.

In the meantime, wanted to make sure you're aware that StorageGRID exposes some metrics via a Prometheus endpoint today. Can you say more about what you want Harvest to provide above and beyond what StorageGRID exposes via its end-point?

[rvalab]

Indeed, I could simply add the StorageGRID Prometheus as a data source in the Harvest's Grafana. Metrics i'm looking for in the StorageGRID is the buckets capacity used. But I realize this info isn't in the Metrics section of the REST API but in the Accounts section :( Maybe it's why it's not in the prometheus database...

[cgrinds]

thanks @rvalab that makes sense - would you be interested in trying to add a new collector that collects the bucket capacity? If so, we could help walk you through that process. @ruanruijuan also ran across these existing examples that may help too

[chrishenzie]

+1, using Harvest to aggregate the metrics exposed by StorageGrid would be great!

[ewilts]

Please add Samsung to the list of customers needing this. A single starting point for dashboards with a consistent interface would be extremely valuable. I envision a folder for the StorageGRID dashboards just like we have for 7-mode, cDot, eg. Performance, capacity, network, power (plus others) by node and by grid are as important in StorageGRID as they are in cDot. And, of course, measuring FabricPool activity overlaps the two for some environments.

[cgrinds]

Copy/pasting more details that @ewilts shared on Slack

I see some grafana graphs in the StorageGRID admin interface but other than starting in the admin interface and clicking through the directory, there is no directory of dashboards like there is in harvest. I don't see anything to get power utilization either so perhaps either the SG team needs to add or Harvest needs a poller to get it (and I don't where from - hardware can be NetApp-branded or total 3rd party, unlike ONTAP)

The caveat on the StorageGRID metrics page: "The tools available on this page are intended for use by technical support. Some features and menu items within these tools are intentionally non-functional."

[iStep2Step]

We would like to have space statistics (Used, Free) for tenants and buckets with corresponding tenant name (not ID )as dashboard. We need this to have a reporting for the Tenant users.

[iStep2Step]

Indeed, I could simply add the StorageGRID Prometheus as a data source in the Harvest's Grafana. Metrics i'm looking for in the StorageGRID is the buckets capacity used. But I realize this info isn't in the Metrics section of the REST API but in the Accounts section :( Maybe it's why it's not in the prometheus database...

I have done exatly this. I get the tenant space usage from Prometheus and the bucket space usage via the Rest API. Unfortunately the tenants are only with ID in Prometheus so I had to do some magic with the variables in the dashboard to get the names.

[cgrinds]

@iStep2Step are your changes/dashboard something you're interested in contributing or is it too specific to your environment?

[iStep2Step]

@iStep2Step are your changes/dashboard something you're interested in contributing or is it too specific to your environment?

I dont think it is specific to my environment but i dont think it is a optimal solution either. I have build this primary for offering a reporting function to our S3 consumer. I can give you the complete JSON or show you what i have done in a call.

[cgrinds]

💯 thanks @iStep2Step! I'll shoot you an email to set something up after the 22.08 release. Appreciate the offer to help here.

[cgrinds]

Copy/pasting more details that Mamoep shared on Discord

You can make StorageGrid internal Prometheus available to external queries since 11.5. https://kb.netapp.com/Advice_and_Troubleshooting/Hybrid_Cloud_Infrastructure/StorageGRID/How_to_enable_external_access_to_Prometheus_for_StorageGRID

[cgrinds]

Copy/pasting more details that shamziman shared on Discord

shamziman — Yesterday at 9:20 AM tenant is probably a better place to start... then buckets -> object count, used capacity ... or even farther up/next to the tree... site capacity, node capacity (bla all the hardware counters) there's a lot of layers, hehe... but not so different from ontap in principle... just cassandra is a bit bigger element than wafl

Chris Grindstaff — Yesterday at 9:46 AM thanks @shamziman if you have a GitHub account, those comments would be a nice addition to the current issue https://github.com/NetApp/harvest/issues/170. If you don't want to bother, I'll add them. Last time I looked closely at StorageGRID, it didn't have a general /metrics endpoint that returns all Prometheus metrics (like what Harvest does). Instead, StorageGRID requires you to query by name. One idea was since StorageGRID already provides open metrics performance data, don't add that to a Harvest collector, but instead focus on the capacity and system health info that it only provides via REST. In other words, build a StorageGRID REST collector

it seems one can access the internal prometheus UI at https://admin_node/metrics/graph ... but i can't really tell you exactly what that means... if you get a test system you can dig into the internal configuration (just a linux box) on the admin node and see what they've done i guess the only advantage of having harvest in the mix is just to have a single point for monitoring, etc ... would be nice to be able to configure custom dashboards and alerts (not a fan of the alarms now... they hang too long) ... i always hope for the convergence of the flexibility of graphana+prometheus and the semi-intelligence of *UM

Chris Grindstaff — Yesterday at 10:05 AM yep that makes sense. looks like StorageGRID bundles Prometheus and Grafana. They ship with a set of dashboards, but if you want those StorageGRID dashboards available in a different Grafana instance, looks like you need to export them from StorageGRID and import into a different instance. If you do that, not sure if it's then possible to change the Prometheus datasource to point to the metrics coming from StorageGRID. I'm not sure yet if StorageGRID exposes the metrics in a way that an external dashboard can use. Maybe you could setup remote writes from the internal Prom instance

shamziman — Yesterday at 10:14 AM we have a number of external SG dashboards as well and I wish I had time to explore making some better ones with the SG sources

[cgrinds]

ReBanana and ! Erik working for Bank Julius Bär & Co AG also requested this collector on Discord.

GitHub usernames: erikgruetter and ReBaunana

[erikgruetter]

ReBanana and ! Erik working for Bank Julius Bär & Co AG also requested this collector on Discord.

GitHub usernames: erikgruetter and ReBaunana

Yes, can confirm the request of @cgrinds. :)

[cgrinds]

We're happy to announce a beta release of the StorageGRID collector is available in the latest nightly build.

This build includes a StorageGRID template and dashboard that collects tenant and bucket data and shows it in a Harvest dashboard (see below). The Tenant variable, at the top of the dashboard, lets you focus on a subset of your tenants.

Please try it out and let us know here or on Discord how it works for you.

To use the new StorageGRID collector, grab nightly and add a new poller to your harvest.yml similar to your other pollers, but with the StorageGrid collector like this:

Pollers:
    pollerName:
        datacenter: rtp
        addr: 1.1.1.1
        username: username
        password: password
        collectors:
            - StorageGrid

[erikgruetter]

@cgrinds This seems amazing! However, is there an ETA when there will be a nightly release for this on NAbox?

[cgrinds]

hi @erikgruetter you should be able to take any Harvest nightly build and deploy inside nabox (assuming there aren't any Harvest changes that invalidate nabox assumptions).

@ybizeul that workflow is supported by nabox, right?

[ybizeul]

Yes that should work. With that said, storagegrid configuration wouldn’t be handled by the web UI of course, it would have to be done manually. I’ll see what I can do to implement storagegrid in a next release though.

[cgrinds]

Verified with 22.11 build commit 982fda55 Dashboard looks great and metrics include tenant id and name

tenant_logical_quota{cluster="VMASGWWEBADM01-TST",datacenter="rtp",id="10698253737774079704",tenant="P14"} 377633636

[rvalab]

Thank you @cgrinds for this feature. I tested harvest-22.11.16-nightly in my environment and it works. Next step for me, trying to recreate same dashboards I have with my Bash script and my Graphite DB to move to Harvest for StorageGRID :)

[cgrinds]

Thanks for the confirmation @rvalab

Feel free to chime in on #1420 if there are other dashboards you need for storagegrid or contribute the one from above and we'll try to add.