opencensus-go-exporter-stackdriver
opencensus-go-exporter-stackdriver copied to clipboard
Retry mechanism?
Moved the issue from https://github.com/census-instrumentation/opencensus-go/issues/766#event-1652116990.
Original issue said:
I am instrumenting an API client for some latency sensitive applications and on turning the sampling rate to always on, which would mimick a server receiving say 50,000 requests per second but with a sampling rate of 1 in 100 so ideally 500 traced QPS I get back thousands of Stackdriver export errors logged on almost a 5 second interval [1]. Perhaps some retry mechanism with exponential backoff or by default using a large buffer as convincing people to using OpenCensus for very traffic applications is bound to happen and it would be worrying to see a bunch of those logs
[1] https://gist.github.com/odeke-em/32cf7359f397a4b93692bcf46109e184 with a sample inlined
$ GOOGLE_APPLICATION_CREDENTIALS=~/creds.json go run main.go
2018/05/27 20:44:47 OpenCensus Stackdriver exporter: failed to upload span: buffer full
2018/05/27 20:44:52 OpenCensus Stackdriver exporter: failed to upload 1126 spans: buffer full
2018/05/27 20:44:57 OpenCensus Stackdriver exporter: failed to upload 1652 spans: buffer full
2018/05/27 20:45:02 OpenCensus Stackdriver exporter: failed to upload 1447 spans: buffer full
2018/05/27 20:45:07 OpenCensus Stackdriver exporter: failed to upload 1598 spans: buffer full
2018/05/27 20:45:12 OpenCensus Stackdriver exporter: failed to upload 925 spans: buffer full
2018/05/27 20:45:17 OpenCensus Stackdriver exporter: failed to upload 1534 spans: buffer full
2018/05/27 20:45:22 OpenCensus Stackdriver exporter: failed to upload 1403 spans: buffer full
2018/05/27 20:45:27 OpenCensus Stackdriver exporter: failed to upload 1034 spans: buffer full
2018/05/27 20:45:32 OpenCensus Stackdriver exporter: failed to upload 1538 spans: buffer full
....
/cc @odeke-em @ramonza