calling Add() with attributeSet results in incorrect values

[pdeva]

Description

When metric.Int64Counter.Add() is called across mutliple threads with an attribute set param, it results in incorrect data. In the graphs shown below, the workload didn't change. However the graph for the call with metric.WithAttributeSet(labels) param shows the workload size constantly growing when it should be flat.

graph with using just ctrOp.Add(ctx, 1)

graph with using ctrOp.Add(ctx, 1, metric.WithAttributeSet(labels))

Environment

• OS: debian
• Architecture: arm64
• Go Version: 1.21
• opentelemetry-go version: 1.19.0

Steps To Reproduce

func init() {
       meter := otel.Meter("redis")
       ctrOp, _ := meter.Int64Counter("req_count")
}

// called from multiple threads
func onRequest() {
     labels := attribute.NewSet( 
         attribute.String("op", "append"),
          attribute.String("table", table),
      )
      ctrOp.Add(ctx, 1, metric.WithAttributeSet(labels))
}

Expected behavior

Graph should remain linear as shown with the usage of ctrOp.Add(ctx, 1) method. the workload didnt change when the withAttributeSet param was added, but the graph changed completely.

[MrAlias]

This doesn't seem to be describing a race condition. Maybe memory consumption?

[pdeva]

well its a bug either way. the workload hasnt changed, but the values reported by otel seem to change once attirbutes are passed in. not sure what memory consumption has to do with this at all.

[MrAlias]

Can you provide a complete setup? With what is provided here I am not able to reproduce your bug. Your code does not compile.

The temorality selection of you meter provider is likely going to be important to know here.

[pdeva]

here is the setup function:

import (
  "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
  "go.opentelemetry.io/otel/sdk/metric"
)
func setupOTelSDK() (shutdown func() error, err error) {
	var shutdownFuncs []func(context.Context) error

	// shutdown calls cleanup functions registered via shutdownFuncs.
	// The errors from the calls are joined.
	// Each registered cleanup will be invoked once.
	shutdown = func() error {
		ctx := context.Background()
		var err error
		for _, fn := range shutdownFuncs {
			err = errors.Join(err, fn(ctx))
		}
		shutdownFuncs = nil
		return err
	}

	exp, err := otlpmetricgrpc.New(context.Background())
	if err != nil {
		return
	}
	meterProvider := metric.NewMeterProvider(metric.WithReader(metric.NewPeriodicReader(exp)))

	shutdownFuncs = append(shutdownFuncs, meterProvider.Shutdown)
	otel.SetMeterProvider(meterProvider)

	return
}

i think thats all the code.

The temorality selection of you meter provider is likely going to be important to know here.

not sure what this means

[pellared]

Please create a minimal, reproducible example.

You can even create a dedicated repository if it would be easier for you.

Also please provide more information e.g. how you create a graph and all that is required to reproduce the issue.

[pdeva]

i did share all the otel related code. the otel data is sent to datadog. you are seeing graphs from datadog in the screenshot

[pellared]

@pdeva Do you plan to add more information that is needed to work on this issue? We need complete (compiling) code and steps that reproduces the issue.