testkube icon indicating copy to clipboard operation
testkube copied to clipboard

Unmarshalling newman results not working in postman-executor

Open RalfWenzel opened this issue 2 years ago • 3 comments

Describe the bug

In https://github.com/kubeshop/testkube/blob/develop/contrib/executor/postman/pkg/runner/newman/metadata.go , Line 67++ :

The RunTimings struct declares ResponseAverage and ResponseSD as float64 but all other timings as int. In my opinion they should be float64 also.

Line 1 of metadata.go says "struct autogenerated from output file" and i think this is the problem. When running newman without --verbose, some values are always 0 (or may be int) so unmarshalling them as int works. But if you run using ``--verbose'' timings have decimals.

Real world example:

    "timings": {
      "responseAverage": 931,
      "responseMin": 931,
      "responseMax": 931,
      "responseSd": 0,
      "dnsAverage": 245.9453359991312,
      "dnsMin": 245.9453359991312,
      "dnsMax": 245.9453359991312,
      "dnsSd": 0,
      "firstByteAverage": 224.44169799983501,
      "firstByteMin": 224.44169799983501,
      "firstByteMax": 224.44169799983501,
      "firstByteSd": 0,
      "started": 1691576601367,
      "completed": 1691576602368
    },

This causes a json: cannot unmarshal number 245.9453359991312 into Go struct field RunTimings.run.timings.dnsAverage of type int error in https://github.com/kubeshop/testkube/blob/develop/contrib/executor/postman/pkg/runner/newman/newman.go, line 126

To Reproduce Steps to reproduce the behavior:

  1. Run a collection with --verbose (Note: output: see real live example above)
  2. See error (Note: See error message in real live example above)

Expected behavior Unmarshalling working -> accept float values

Version / Cluster

  • Which testkube version? -> latest
  • What Kubernetes cluster? -> Azure AKS
  • What Kubernetes version? -> 1.25.6

Screenshots image

Additional context n/a

RalfWenzel avatar Aug 09 '23 12:08 RalfWenzel

Hey, @RalfWenzel Can you please provide an example of postman collection for testing?

vsukhin avatar Aug 11 '23 17:08 vsukhin

@RalfWenzel we need an example of this test to reproduce the issue to fix it. if you could provide please.

TheBrunoLopes avatar Sep 19 '23 10:09 TheBrunoLopes

@TheBrunoLopes , hey, sorry, i was offline for a couple of weeks. Will prepare this now.

RalfWenzel avatar Sep 20 '23 14:09 RalfWenzel