exporter/stackdriver: error posting too frequent when >1 pod reports same metrics,

[rghetia]

On GKE deployment if more than one pod reports same metrics then SD backend rejects data points with the reason ""rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric. "

This is because there is no label that differentiates data points from two different nodes. Here is a sample tracez page from an ocagent.

Span Name: contrib.go.opencensus.io/exporter/stackdriver.uploadMetrics

10 Requests

When                       Elapsed (sec)
----------------------------------------
2019/04/05-20:24:48.505738      .224350 trace_id: 4dca2a35177c0f21c30522b54722b337 span_id: ef227ffa88b5fb70
                                        Status{canonicalCode=UNKNOWN, description="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/opencensus/grpc.io/server/received_bytes_per_rpc, Timestamps: {Youngest Existing: '2019/04/05-13:24:44.182', New: '2019/04/05-13:24:44.504'}}: timeSeries[1]; One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/opencensus/grpc.io/server/sent_bytes_per_rpc, Timestamps: {Youngest Existing: '2019/04/05-13:24:44.182', New: '2019/04/05-13:24:44.504'}}: timeSeries[2]"}
2019/04/05-20:25:48.507615      .259883 trace_id: de00c24549ebb805bf823ea48dbf707b span_id: e969bae9e354d838
                                        Status{canonicalCode=UNKNOWN, description="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/opencensus/grpc.io/server/completed_rpcs, Timestamps: {Youngest Existing: '2019/04/05-13:25:44.182', New: '2019/04/05-13:25:44.504'}}: timeSeries[1]"}
2019/04/05-20:14:48.539989      .292571 trace_id: 8036da2a6e7a5eb3f2dba4dd06a74949 span_id: 219fc31ed27e0a09
                                        Status{canonicalCode=UNKNOWN, description="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/opencensus/grpc.io/server/completed_rpcs, Timestamps: {Youngest Existing: '2019/04/05-13:14:44.182', New: '2019/04/05-13:14:44.504'}}: timeSeries[0]"}
2019/04/05-20:15:48.505581      .286585 trace_id: 4350702dda33c88e992293d216ca6c82 span_id: aef36197d454ff24
                                        Status{canonicalCode=UNKNOWN, description="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/opencensus/grpc.io/server/sent_bytes_per_rpc, Timestamps: {Youngest Existing: '2019/04/05-13:15:44.182', New: '2019/04/05-13:15:44.504'}}: timeSeries[0]"}
2019/04/05-20:17:48.239556      .267271 trace_id: fe7194ec61041fc1981de2549c407457 span_id: e057eb6e37c04e57
                                        Status{canonicalCode=UNKNOWN, description="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/opencensus/grpc.io/server/completed_rpcs, Timestamps: {Youngest Existing: '2019/04/05-13:16:44.182', New: '2019/04/05-13:16:44.504'}}: timeSeries[1]"}
2019/04/05-20:18:48.339702      .390439 trace_id: e01604e849b9c14c8f54b088042b98a7 span_id: e8fe9f8a830df6cc
                                        Status{canonicalCode=UNKNOWN, description="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/opencensus/grpc.io/server/completed_rpcs, Timestamps: {Youngest Existing: '2019/04/05-13:17:44.182', New: '2019/04/05-13:17:44.540'}}: timeSeries[0]"}
2019/04/05-20:19:48.505589      .244092 trace_id: fa925922c32ae224eac581e5a477c827 span_id: 9e199b39490bd106
                                        Status{canonicalCode=UNKNOWN, description="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/opencensus/grpc.io/server/received_bytes_per_rpc, Timestamps: {Youngest Existing: '2019/04/05-13:19:44.183', New: '2019/04/05-13:19:44.504'}}: timeSeries[0]"}
2019/04/05-20:20:48.539674      .100707 trace_id: b2118554f204698b1b55457dc42a7877 span_id: d636e92151d74371
                                        Status{canonicalCode=UNKNOWN, description="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: <nil>"}
2019/04/05-20:22:48.183572      .328955 trace_id: ebcdc86357445fb4783d3f71ef267c53 span_id: 2706fc750a59de74
                                        Status{canonicalCode=UNKNOWN, description="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/opencensus/grpc.io/server/received_bytes_per_rpc, Timestamps: {Youngest Existing: '2019/04/05-13:21:44.182', New: '2019/04/05-13:21:44.505'}}: timeSeries[0]"}
2019/04/05-20:23:48.240196      .377988 trace_id: 25b885a0b5008fef7fb7e429dc851fec span_id: ff3617c59a27baf5
                                        Status{canonicalCode=UNKNOWN, description="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/opencensus/grpc.io/server/server_latency, Timestamps: {Youngest Existing: '2019/04/05-13:22:44.182', New: '2019/04/05-13:22:44.504'}}: timeSeries[0]"}

[flands]

@rghetia what is the latest on this?

[rghetia]

@rghetia what is the latest on this?

It is still not fixed.