null value being converted to unicode

[sandiezarate]

^^ Provide a general summary of the issue in the title above. ^^

Description

Describe the problem you're encountering. TIP: Do NOT share sensitive information, whether personal, proprietary, or otherwise!

I'm using nri-flex and then calling the curl command to get some metrics from MongoDB In the output, the null' value is being converted to unicode characters ("\u003cnil\u003e") or equivalent to "<nil>"

Expected Behavior

Tell us what you expected to happen.

I expected to change the null value to 0.

Troubleshooting or NR Diag results

Provide any other relevant log data. TIP: Scrub logs and diagnostic information for sensitive information

Steps to Reproduce

Please be as specific as possible. TIP: Link a sample application that demonstrates the issue.

I based my script on: mongodb-atlas-curl.yml Portion of the code:

        - run: curl -s -u "<xxxx:yyyy>" --digest "https://cloud.mongodb.com/api/atlas/v1.0/groups/#####/processes/######:#####/measurements?granularity=PT1M&period=PT5M&m=OPLOG_REPLICATION_LAG_TIME&pretty=true"
          timeout: 30000
      jq: .measurements | map(select(.dataPoints[0].value != null)) | map({(.name):.dataPoints[0].value}) | add

Portion of the output:

                           {
                                    "event_type": "TMP_MongoDbAtlasProcessMetricsSample",
                                    "integration_name": "com.newrelic.nri-flex",
                                    "integration_version": "1.4.3",
                                    "name": "OPLOG_REPLICATION_LAG_TIME",
                                    "timestamp": "2022-10-20T04:21:58Z",
                                    "units": "SECONDS",
                                    "value": "\u003cnil\u003e"
                            },
                            {
                                    "event_type": "TMP_MongoDbAtlasProcessMetricsSample",
                                    "integration_name": "com.newrelic.nri-flex",
                                    "integration_version": "1.4.3",
                                    "name": "OPLOG_REPLICATION_LAG_TIME",
                                    "timestamp": "2022-10-20T04:22:58Z",
                                    "units": "SECONDS",
                                    "value": 0
                            },

Error message:

jq: failed to process api=MongoDbAtlasProcessMetrics error="" jq=".[0].measurements | map(select(.dataPoints[0].value != null)) | map ({(.name):.dataPoints[0].value}) | add"

Your Environment

Include as many relevant details about your environment as possible including the running version of New Relic software and any relevant configurations.

Additional context

Add any other context about the problem here. For example, relevant community posts or support tickets.

For Maintainers Only or Hero Triaging this bug

Suggested Priority (P1,P2,P3,P4,P5): Suggested T-Shirt size (S, M, L, XL, Unknown):

[workato-integration[bot]]

https://issues.newrelic.com/browse/NEWRELIC-4729

[rubenruizdegauna]

Hi @sandiezarate , can you add the raw output of the curl command (replacing any sensible data by *****) with and w/o "pretty" parameter?

curl -s -u "<xxxx:yyyy>" --digest "https://cloud.mongodb.com/api/atlas/v1.0/groups/#####/processes/######:#####/measurements?granularity=PT1M&period=PT5M&m=OPLOG_REPLICATION_LAG_TIME&pretty=true

AND

curl -s -u "<xxxx:yyyy>" --digest "https://cloud.mongodb.com/api/atlas/v1.0/groups/#####/processes/######:#####/measurements?granularity=PT1M&period=PT5M&m=OPLOG_REPLICATION_LAG_TIME&pretty=false

This way we can see the value before being parsed by jq.

Thanks!

[sandiezarate]

pretty_true.txt pretty_false.txt

[sandiezarate]

DEBU[0002] jq: failed to process api=MongoDbAtlasProcessMetrics error="<nil>" jq=".[] | .measurements | map(select(.dataPoints[0].value != null)) | map({(.name):.dataPoints[0].value}) | add"

[sandiezarate]

I see the error message: "jq: failed to process" in this code: internal/processor/data.go

[rubenruizdegauna]

Hi @sandiezarate ,

I have run the nri-flex with the output you provided and it parses it correctly.

name: Test
apis:
  - name: MongoDbAtlasProcess
    event_type: MongoDbAtlasProcessSample
    commands:
      - run: curl -s http://localhost:8000/pretty_false.json
    jq: .measurements | map(select(.dataPoints[0].value != null)) | map({(.name):.dataPoints[0].value})|add

Result:

{"name":"com.newrelic.nri-flex","protocol_version":"3","integration_version":"Unknown-SNAPSHOT","data":[{"metrics":[{"OPLOG_REPLICATION_LAG_TIME":0,"event_type":"MongoDbAtlasProcessSample","integration_name":"com.newrelic.nri-flex","integration_version":"Unknown-SNAPSHOT"},{"event_type":"flexStatusSample","flex.Hostname":"XLQJ24H4X4","flex.IntegrationVersion":"Unknown-SNAPSHOT","flex.counter.ConfigsProcessed":1,"flex.counter.EventCount":1,"flex.counter.EventDropCount":0,"flex.counter.MongoDbAtlasProcessSample":1,"flex.time.elapsedMs":30,"flex.time.endMs":1667469270682,"flex.time.startMs":1667469270652}],"inventory":{},"events":[]}]}

Are you using the latest version?

[sandiezarate]

Yes, I am using the version 1.5.3. The direct output of the curl command from nri-flex is causing the error because of the 'null' values being converted to '<nil>'

[rubenruizdegauna]

This is unexpected. In the config above I've simulated the http call you are making to mongodb, so I'm outputting the same content that you get. And I'm using the same configuration and it just discards the null value and get the 0 one.

From what you mention your solution is based on the example and the configuration you added is just a part. Can you try the configuration I added here pointing to the mongo and see if it works? If this works then it might be something else wrong with the configuration.

The pretty_false.json is the content you attached which contains the null value and doesn't return any error.