UnhandledPromiseRejectionWarning: Error: One or more TimeSeries could not be written

[DanielWilliamClarke]

Please answer these questions before submitting a bug report.

What version of OpenCensus are you using?

0.0.18

What version of Node are you using?

10.15.3

What did you do?

I create a simple time series view, I use the timeseries metric descriptor my-app/app-latency (for example)

// Create Metric
const latency = globalStats.createMeasureDouble(
    'my-app/AppLatency', MeasureUnit.MS, 'The service response latencies');

// Create View
const viewLatency = globalStats.createView(
    'my-app/app-latency', // TIMESERIES METRIC DESCRIPTOR NAME
    latency,
    AggregationType.DISTRIBUTION,
    [],
    'The distribution of the latencies',
    [0, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000],
);

// Register
globalStats.registerView(viewLatency);

// Record an event
globalStats.record([{
    measure: viewLatency,
    value: hrend[0] * 1000 + hrend[1] / 1000000,
}]);

What did you expect to see?

I expect to see no UnhandledPromiseRejectionWarning error on flush to stackdriver, I expect to see the metric type to be a valid URL with domain and non empty path (while debugging)

custom.googleapis.com/opencensus/my-app/app-latency

What did you see instead?

On flush to stackdriver, I see the following errors:

(node:5520) UnhandledPromiseRejectionWarning: Error: One or more TimeSeries could not be written:
type Field timeSeries[0].metric.type had an invalid value of "custom.googleapis.com\opencensus\my-app\app-latency":
    The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0];                    

    at Gaxios.request (C:\Workspace\my-app-service\node_modules\gaxios\build\src\gaxatchios.js:70:23)
    at process._tickCallback (internal/process/next_tick.js:68:7)

(node:5520) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either side of an async function without a catch block, or by rejecting a promise which was not handled with .on id: 1)

(node:5520) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, s that are not handled will terminate the Node.js process with a non-zero exit code.

Upon debugging the issue, I can see that the metric type error given above is given as the value is transformed into the following:

custom.googleapis.com\opencensus\my-app\app-latency

I traced the issue to the following method inside:

exporter-stackdriver/build/src/stackdriver-monitoring-utils.js

/** Creates Metric type. */
function createMetricType(name, metricPrefix) {
    return path.join(metricPrefix, name);
}

I can see that when refering to the node js docs on path.join

The path.join() method join all given path segments together using the platform specific separator as a delimiter, then normalizes the resulting path.

It seems that if the metric type is meant to a valid URL (for exporting to stackdriver) then using path is not the correct solution to combine the prefix and descriptor.

This error occurs in our windows development environments only.

Additional context

Add any other context about the problem here.

[mayurkale22]

Thanks for reporting this!

Is it possible to provide a complete dump of generated TimeSeries json? At first glance, metric.type (custom.googleapis.com/opencensus/my-app/app-latency) looks correct. Maybe, some issue with the TimeSeries value.

[patwhite]

I'm also seeing this, quite verbosely in fact! It could also be related to the same issue the python and java opencensus implementations had: https://github.com/census-instrumentation/opencensus-python/issues/350 https://github.com/census-instrumentation/opencensus-java/issues/865

Which looked like they had to do with flushing. I'm running on cloud run, so that too may be causing issues.

Here's the message, I see it every 5 minutes or so in the log: One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older start time than the most recent point.: timeSeries[0-3]

I don't know if this is related to the metric the user mentioned up above, or every metric like in the python and java issues, I haven't done any debugging.

[mayurkale22]

Is it possible to provide some recipe to reproduce this? I will try to debug this at my end.

[patwhite]

It's consistently happening for us, we usually have two containers running, and I'd say almost always are submitting more than 1 metric at a time. I have the timeout set at 90 seconds.

On Tue, Dec 3, 2019 at 11:02 AM Mayur Kale [email protected] wrote:

Is it possible to provide some recipe to reproduce this? I will try to debug this at my end.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/census-instrumentation/opencensus-node/issues/694?email_source=notifications&email_token=ABGISN4AKG6DAZC2CXX35HDQW2ULHA5CNFSM4JAPXE42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF2ON3A#issuecomment-561309420, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGISN2FJF6F5YWK3A2ZQCLQW2ULHANCNFSM4JAPXE4Q .

[patwhite]

@mayurkale22 Were you able to repro this? I’d be happy to give you a dump, is there’s a way to turn on verbose logging through env variables?

[mayurkale22]

AFAIK this scenario possible in distributed system when multiple clients writing to the same metric. Ideally, MonitoredResource (https://cloud.google.com/monitoring/api/resources) should address this problem (if detected properly). The Stackdriver Exporter tries to find accurate MonitoredResource but if the expected labels are missing then it uses a global Stackdriver monitored resource with no labels.

Could you please try OC_ env variable approach (mentioned here)? Also, please confirm whether you're seeing on single client or multiple clients setup?