F5Networks
OpenTelemetry exporter fails with message Cannot read property 'Resolver' of undefined
Environment
- Telemetry Streaming Version: 1.36.0-1
- BIG-IP Version: 17.1.1.4
Summary
Opentelemetry_Exporter type of consumer fails to send the collected metrics. Verifying by tcpdump on the target shows that no attempt to open the tcp connection to send the report is made. In the /var/log/restnoded/restnoded.log file this line is printed:
Thu, 26 Sep 2024 08:25:09 GMT - severe: [telemetry.service.ConsumersService.OpenTelemetry_Exporter.f5telemetry_default::SRV01_OTEL_GRPC_NOSSL] error: Cannot read property 'Resolver' of undefined
Impact is that I cannot use gRPC to send Opentelemetry reports from BIG-IP.
Steps To Reproduce
Steps to reproduce the behavior:
- Submit the following declaration:
{
"$schema": "https://raw.githubusercontent.com/F5Networks/f5-telemetry-streaming/master/src/schema/latest/base_schema.json",
"class": "Telemetry",
"schemaVersion": "1.36.0",
"controls": {
"class": "Controls",
"logLevel": "debug",
"debug": false,
"memoryMonitor": {
"logFrequency": 60
},
"memoryThresholdPercent": 90
},
"Listener": {
"class": "Telemetry_Listener",
"port": 6514,
"enable": true,
"trace": false,
"match": "",
"actions": [
{
"setTag": {
"tenant": "`T`",
"application": "`A`"
},
"enable": true
}
]
},
"defaultPoller": {
"class": "Telemetry_System_Poller",
"interval": 60,
"host": "localhost",
"port": 8100,
"protocol": "http",
"allowSelfSignedCert": true,
"enable": true,
"workers": 5,
"chunkSize": 30
},
"telemetrySystem": {
"class": "Telemetry_System",
"enable": true,
"systemPoller": [
"defaultPoller"
],
"host": "localhost",
"port": 8100,
"protocol": "http",
"enableHostConnectivityCheck": false,
"allowSelfSignedCert": true
},
"SRV01_OTEL_GRPC_NOSSL": {
"class": "Telemetry_Consumer",
"type": "OpenTelemetry_Exporter",
"enable": true,
"trace": true,
"host": "IP-address-redacted",
"port": 4317,
"exporter": "grpc",
"convertBooleansToMetrics": false,
"useSSL": false
}
}
- Declaration is accepted, response is 200 OK:
{
"message": "success",
"declaration": {
...
- Wait for the poller to collect data and observe the restnoded.log file for result. Also, verify with tcpdump if any connection is attempted to the consumer.
Expected Behavior
The consumer of type OpenTelemetry_Exporter would be succesful in sending telemetry reports using gRPC.
Actual Behavior
No report is sent out. Based on logs, data is collected, but sending to the consumer fails even before a TCP connection is attempted:
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Starting polling cycle
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully completed "WAITING" step
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Transitioning from step "WAITING" to "COLLECT"
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller.Loader[localhost]] Task queue "TaskQueue_9dd6d" configured with concurrency = 5
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller.context.Collector] Starting stats collection
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller.context.Collector] Task queue "TaskQueue_4667f" configured with concurrency = 5
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller.stats.Collector] Starting stats collection
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller.stats.Collector] Task queue "TaskQueue_918df" configured with concurrency = 5
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully collected stats
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully completed "COLLECT" step
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Transitioning from step "COLLECT" to "SEND_REPORT"
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully processed stats report
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully completed "SEND_REPORT" step
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Transitioning from step "SEND_REPORT" to "DONE"
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully collected and processed stats. Polling Cycle duration - 0 sec.
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully completed "DONE" step
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Transitioning from step "DONE" to "SCHEDULE"
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Next polling cycle starts on 2024-09-26T08:26:09.029Z (in 59 s.)
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully scheduled next execution date
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Successfully completed "SCHEDULE" step
Thu, 26 Sep 2024 08:25:09 GMT - finest: [telemetry.service.SystemPollerService.Poller[f5telemetry_default::telemetrySystem::defaultPoller].Poller] Transitioning from step "SCHEDULE" to "WAITING"
Thu, 26 Sep 2024 08:25:09 GMT - severe: [telemetry.service.ConsumersService.OpenTelemetry_Exporter.f5telemetry_default::SRV01_OTEL_GRPC_NOSSL] error: Cannot read property 'Resolver' of undefined
Hi @jussisjostrom,
New versions of OTEL library can't be used because node.js is too old and OTEL libs has been updated and it fails now. So it even fails on the version that worked in the past: the server side was updated and as result the client side (TS) tries to trigger the code that even does not exist in the library or does not work on node.js 8.11.1
@jussisjostrom
you can try http/protobuf instead of grpc if you server configured to accept it
Same problem here after upgrading to Telemetry 1.37
"f5-collector": { "class": "Telemetry_Consumer", "type": "OpenTelemetry_Exporter", "host": "10.41.85.46", "port": 4317, "trace": true, "enable": true, "headers": [ { "name": "x-access-token", "value": "YOUR_TOKEN" } ], "convertBooleansToMetrics": true, "exporter": "grpc", "useSSL": false, "allowSelfSignedCert": true },
What's the plan? should we revert back to previous version or will someone fix issue?
Hi @PatricDahl
New versions of OTEL library can't be used because node.js is too old and OTEL libs has been updated and it fails now. So it even fails on the version that worked in the past: the server side was updated and as result the client side (TS) tries to trigger the code that even does not exist in the library or does not work on node.js 8.11.1.
Can you try http/protobuf instead of grpc if you server configured to accept it?
