harvest Harvest should collect filesystem analytics IOPs and xput

Thanks for the feature request Rusty Brown!

Reference

https://library.netapp.com/ecmdocs/ECMLP2884821/html/#docs-storage-storage_volumes_{volume.uuid}_top-metrics_files

May 30 '23 14:05 cgrinds

The following endpoints are required to gather IOPS and throughput data for each volume. By default, we are dealing with 20 volumes sorted by size. If we call them every 3 minutes that might not be as effective, given that these APIs do not support raw counters that would allow us to process the data. Therefore, the data we collect might not be particularly useful to display.

Moreover, we would need to make four calls per instance. This means that, by default, we would have to invoke the ONTAP 80 times every 3 minutes according to our config schedule.

Here are the endpoints:

/api/storage/volumes/b8243f95-dd15-11ed-9baf-00a098d390f2/top-metrics/files?top_metric=iops.read&fields=iops.error%2Csvm%2Cvolume%2Cpath&return_timeout=120 /api/storage/volumes/b8243f95-dd15-11ed-9baf-00a098d390f2/top-metrics/files?top_metric=iops.write&fields=iops.error%2Csvm%2Cvolume%2Cpath&return_timeout=120 /api/storage/volumes/b8243f95-dd15-11ed-9baf-00a098d390f2/top-metrics/files?top_metric=throughput.read&fields=throughput.error%2Csvm%2Cvolume%2Cpath&return_timeout=120 /api/storage/volumes/b8243f95-dd15-11ed-9baf-00a098d390f2/top-metrics/files?top_metric=throughput.write&fields=throughput.error%2Csvm%2Cvolume%2Cpath&return_timeout=120

We'll need this information via perf collector which is currently unavailable. Moving this out of 23.11.

Oct 10 '23 08:10 rahulguptajss

We can probably use top_file object in ZapiPerf to collect this information. There is no such object in RestPerf currently.

Nov 28 '23 06:11 rahulguptajss

We can probably use top_file object in ZapiPerf to collect this information. There is no such object in RestPerf currently.

ZapiPerf collector doesn't support this object. This approach will not work.

This command does not support the 'top_client' object and other statistically tracked objects. Please use the "statistics start", "statistics stop", and "statistics show" commands.

Nov 28 '23 08:11 rahulguptajss

We can probably use top_file object in ZapiPerf to collect this information. There is no such object in RestPerf currently.

ZapiPerf collector doesn't support this object. This approach will not work.
This command does not support the 'top_client' object and other statistically tracked objects. Please use the "statistics start", "statistics stop", and "statistics show" commands.

The Harvest CLI command bin/harvest zapi -p sar show data --object top_file works. ZapiPerf collector encounters an error due to a failure with perf-object-instance-list-info. The Harvest CLI avoids this issue by passing * as instances to perf-object-get-instances and not invoking perf-object-instance-list-info. To make objects like top_file and top_client work, we have two solutions:

Modify the ZapiPerf collector to pass * as instances to perf-object-get-instances for such objects.
Investigate the use of a private CLI.

Dec 14 '23 12:12 rahulguptajss