Resource: Sanitize the "location" label for k8s resource.

[songy23]

~~For the "location" label of k8s resource, we used to get the value from GcpMetadataConfig.getZone() and pass it directly to Stackdriver Monitoring, and it worked fine previously. However it looks like the format of GCP metadata Zone has changed recently: (https://github.com/googleapis/gax-dotnet/issues/233):~~

• ~~Before: "us-central1-d"~~
• ~~Now: "projects/.../zones/us-central1-d"~~

~~If we directly passed the new value to Stackdriver, an error will be returned saying that com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: Unrecognized region or location.: ....~~

~~This can be fixed by only extracting the location value from the new label.~~

[mayurkale22]

Looks like we already have a logic to sanitize the metadata value for the zone. The original value has the following format: projects/[NUMERIC_PROJECT_ID]/zones/[ZONE]. https://github.com/census-instrumentation/opencensus-java/blob/master/contrib/monitored_resource_util/src/main/java/io/opencensus/contrib/monitoredresource/util/GcpMetadataConfig.java#L51-L53 -> with this it will return [ZONE] value.

[bogdandrutu]

What version did the user used when the problem happened?

[mayurkale22]

What version did the user used when the problem happened?

opencensus-java 0.18.0 Edit: when running a docker image within k8s on GCP.

[songy23]

@samhaink Could you try ssh to one of the container and see if you're able to retrieve the location info? Use wget --header "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/instance/zone".

@mayurkale22 has tried to reproduce this error but it looks like everything is working fine with our GKE instance.

[samhaink]

@songy23 Sorry for the delay.

root@xxxxxx-7db496f4b9-5zxzv:/# wget --header "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/instance/zone"
--2019-01-30 00:32:56-- http://metadata.google.internal/computeMetadata/v1/instance/instance/zone
Resolving metadata.google.internal (metadata.google.internal)... 169.254.169.254
Connecting to metadata.google.internal (metadata.google.internal)|169.254.169.254|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2019-01-30 00:32:56 ERROR 404: Not Found.

If I drop the duplicate 'instance/instance', I receive the following

--2019-01-30 07:34:56--  http://169.254.169.254/computeMetadata/v1/instance/zone
Connecting to 169.254.169.254:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/text]
Saving to: ‘zone'

The contents of that is 'projects/99999999/zones/us-east1-c'

Just for reference, this was the MonitoredResource object I had to create in order to get around the "Unrecognized region or location" error.

MonitoredResource myResource = MonitoredResource.newBuilder()
                        .setType("k8s_container")
                        .putLabels("project_id", "xxxx")
                        .putLabels("location", "us-central1-f")
                        .putLabels("pod_name", "xxxxxx-75fd9976d7-7gdl8")
                        .putLabels("container_name", "xxxxx")
                        .putLabels("cluster_name", "xxx-ci")
                        .putLabels("namespace_name", "xxxxx-dev")
                        .build();

                StackdriverStatsExporter.createAndRegister(
                        StackdriverStatsConfiguration.builder()
                                .setCredentials(GoogleCredentials.fromStream(credentialsJSON))
                                .setMonitoredResource(myResource)
                                .setProjectId("xxxxx")
                                .build());

[songy23]

@samhaink Apologies, the second url is correct. Looks like you're able to get the zone string (projects/*/zones/us-east1-c). With our resource util, "us-east1-c" should be auto-detected and added as "location" in the resource labels (https://github.com/census-instrumentation/opencensus-java/blob/master/contrib/monitored_resource_util/src/main/java/io/opencensus/contrib/monitoredresource/util/GcpMetadataConfig.java#L46-L55).

This looks strange. We'll try to reproduce this error again.