Spring Cloud Contract Generates Weak Tests

[DhavalShewale]

Producer side test cases generated by Spring Cloud Contract have weak assertions which can also pass even while it is behaving incorrectly.

Sharing a sample contract to elaborate on my concern.

Contract

request:
  method: GET
  url: /staticData/getAnswers
response:
  status: 200
  body:
    - "value": "0"
      "displayText": "No"
    - "value": "1"
      "displayText": "Yes"
  headers:
    Content-Type: application/json

Generated Test

...
DocumentContext parsedJson = JsonPath.parse(response.getBody().asString());
assertThatJson(parsedJson).array().contains("['value']").isEqualTo("0");
assertThatJson(parsedJson).array().contains("['displayText']").isEqualTo("No");
assertThatJson(parsedJson).array().contains("['value']").isEqualTo("1");
assertThatJson(parsedJson).array().contains("['displayText']").isEqualTo("Yes");
...

The above generated test would also pass if the producer returned any of the below incorrect outputs.

[
  {
    "value": "0",
    "displayText": "Yes"
  },
  {
    "value": "1",
    "displayText": "No"
  }
]

[
  {
    "value": "0",
    "displayText": "No"
  },
  {
    "value": "1",
    "displayText": "Yes"
  }, 
  {
    "value": "-1",
    "displayText": "NA"
  }, 
]

Here I would like to highlight two concerns:

• The test can be false successful by cross comparing the data as highlighted in example 1
• The test can be false successful even when additional unexpected elements are returned

Also, I tried using spring.cloud.contract.verifier.assert.size and assertJsonSize property but they had no effect on the generated test classes so it seems to be fully ignored.

Generated test classes should look more like the below code to avoid such false success scenarios

DocumentContext parsedJson = JsonPath.parse(response.getBody().asString());
assertThatJson(parsedJson).elementWithIndex(0).field("['value']").isEqualTo(0);
assertThatJson(parsedJson).elementWithIndex(0).field("['displayText']").isEqualTo("No");
assertThatJson(parsedJson).elementWithIndex(1).field("['value']").isEqualTo(1);
assertThatJson(parsedJson).elementWithIndex(1).field("[displayText']").isEqualTo("Yes");
assertThatJson(parsedJson).array().hasSize(2);

This code was tested on Java 11 with Spring Boot Version 2.7.13 and Spring Cloud Contract Version 3.1.8

[DhavalShewale]

@marcingrzejszczak: Tks for acknowledging the request and tagging it as an enhancement.

I wanted to enquire whether this can be taken up in the near future or this is currently at the bottom of the stack pending prioritization.