Getting `Duplicate TimeSeries encountered` error logs on sending opentelemtry measures and empty resources info

[moshevi]

Hi. Running kubeflow pipeline on google cloud with python package using these:

opentelemetry-api>=1.18.0,
opentelemetry-sdk>=1.18.0,
opentelemetry_resourcedetector_process>=0.3.0'
opentelemetry-propagator-gcp>=1.5.0,
opentelemetry-resourcedetector-gcp>=1.5.0a0,
opentelemetry-exporter-gcp-monitoring>=1.5.0a0,
opentelemetry_resourcedetector_kubernetes>=0.3.0

1. I'm getting an opentelemtry error: One or more TimeSeries could not be written: Field timeSeries[3] had an invalid value: Duplicate TimeSeries encountered. Only one point can be written per TimeSeries per request.; Field timeSeries[1] had an invalid value: Duplicate TimeSeries encountered. Only one point can be written per TimeSeries per request.
2. I don't see any resource info regarding my metric on google cloud metric explorer

I'm init my metrics as follows:

       __resources = get_aggregated_resources(
           [
               KubernetesResourceDetector(),
               GoogleCloudResourceDetector(),
               ProcessResourceDetector(),
           ],
           timeout=60,
       )

       metrics.set_meter_provider(
          meterProvider(
               views=[
                   view.View(
                       instrument_name='*.duration',
                       aggregation=__view.ExplicitBucketHistogramAggregation((
                           0, 25, 50, 75, 100, 200, 400, 600, 800, 1000, 2000,
                           4000, 6000, 10000, 20000, 30000, 60000, 120000))
                   )
               ],
               metric_readers=[
                   PeriodicExportingMetricReader(
                       CloudMonitoringMetricsExporter(
                           add_unique_identifier=True,
                           prefix='custom.googleapis.com/companyName/my-shared-libary',
                       ),
                   ),
               ],
               resource=__resources
           )
       )

       counter=metrics.get_meter('MeterName').create_counter(name='counter_metric_name', unit='1')
       hist=metrics.get_meter('MeterName').create_histogram(name='hist_metric_name',unit='ms')

And sending metrics as follows:

labels={'product':'productA','status':'OK'}
counter.add(1, labels)
duration = round((datetime.now() - startTime).total_seconds() * 1000)
hist.record(duration, labels)

The measure seems ok on the google cloud metric explorer but I wonder why I see this error on logs and why I don't see the resources data there, any idea ?

[aabmass]

@moshevi are you using gunicorn or uwsgi? We sometimes see this error if your process is forking.

[aabmass]

@moshevi is this still an issue you're facing?

[moshevi]

@aabmass Sorry, missed your message? what do you mean by gunicorn or uwsgi, in which python package? And yes I still get this error. Do you see any issue in my code that can cause this error ?

[aabmass]

This is most likely a resource detection issue. The error "Duplicate TimeSeries encountered" usually means multiple copies of your application are writing the same exact metric "stream" (all the same resource and metric labels). Gunicorn and uwsgi are frequently used to run python applications with multiple worker processes and there are some caveats with OpenTelemetry. Do you know if your kubeflow is using Gunicorn or uwsgi?