spanner: support custom meterprovider for builtin metrics

[tyrone-anz]

Is your feature request related to a problem? Please describe.

Spanner client is sending builtin/native metrics directly to GCP by default.

For services that have no permission, the metric export results to error log and span with following message.

Permission monitoring.timeSeries.create denied (or the resource may not exist).

For the above noise to go away, the native metrics can be disabled (SPANNER_DISABLE_BUILTIN_METRICS=true) or simply give service the missing permission. However, this is always something to keep in mind for a new service using spanner.

Describe the solution you'd like

Use the ClientConfig.OpenTelemetryMeterProvider from the config

Code affected. https://github.com/googleapis/google-cloud-go/blob/5b59819e2d603ef55c4cf056b70af6a08d335373/spanner/client.go#L496-L511

Describe alternatives you've considered

• Allow a param that sends builtin metrics to the global meter provider otel.GetMeterProvider()

Additional context

Add any other context or screenshots about the feature request here.

[tyrone-anz]

Just wanted to bump this one. I am happy to raise PR for if okay to support.

[ymmt2005]

Hello,

We experienced the same issue. The issue was very confusing because we configured a global meter provider with otel.SetMeterProvider.

It'd be great if the native metrics could use the global meter provider if set.

[surbhigarg92]

@tyrone-anz @ymmt2005 These built-in metrics are enabled by default in the Spanner client library. They are primarily intended for use by Spanner engineers and support teams to help diagnose and resolve customer issues more efficiently. To make these metrics visible in Google Cloud Monitoring, we use our own MeterProvider with out custom exporter to export them.

We could consider a feature request to allow customers to configure an additional MeterProvider alongside ours. This would enable the metrics to be exported to both the internal and customer-specified destinations.

Regarding Permission issue , the monitoring.timeseries.create permission is included in Spanner IAM Roles to help customers avoid this issue . But if you are using Custom Roles, then this permission needs to be added explicitly. Refer https://cloud.google.com/spanner/docs/view-manage-client-side-metrics#access-client-side-metrics

[ymmt2005]

@surbhigarg92 Thank you for the info.

Actually, we give our programs roles/spanner.databaseUser but only for the databases used by the program. The inclusion of monitoring.timeseries.create did not help because for it to work, the Spanner role should be given to the project rather than each database.

[apstndb]

I came across this issue for a different reason.

• I want to use OpenTelemetry's ManualReader to directly read GFE/AFE latency from the client application.
• Both GFE and AFE latency are available in the Native Metrics, but when using metrics from OpenTelemetry's MeterProvider (as defined in spanner/ot_metrics.go), only GFE latency is present—AFE latency is missing.

For this reason, I would also like to request that metrics be provided via MeterProvider, as is the main topic of this issue.