Tavern 2.0.0a2 False Negatives

[KenStorey]

Tavern 2.0.0a2 reports erroneous failures.

/tavern_tests/test_gdcp_display_priorities.tavern.yaml::Volume, no adjustment ______________Format variables:
  hmi_app_connect_topic = 'hmi/app_connect'
  high_pri_app = 'MCX_Client'
  hmi_app_connect_response_topic = 'hmi/app_connect_response'
  high_pri_app = 'MCX_Client'

Source test stage (line 37):
  - name: Step 1 - Connect high priority app

    mqtt_publish:
      topic: "{hmi_app_connect_topic}"
      json:
        application: "{high_pri_app}"
        application_ref: 1000

    mqtt_response:
      topic: "{hmi_app_connect_response_topic}"
      json:
        application: "{high_pri_app}"
        application_ref: 1000
        result_code: 0
      timeout: 1


Formatted stage:
  mqtt_publish:
    json:
      application: 'MCX_Client'
      application_ref: 1000
    topic: 'hmi/app_connect'
  mqtt_response:
    json:
      application: 'MCX_Client'
      application_ref: 1000
      result_code: 0
    timeout: 1
    topic: 'hmi/app_connect_response'
  name: Step 1 - Connect high priority app

Errors:
E   tavern.util.exceptions.TestFailError: Test 'Step 1 - Connect high priority app' failed:
    - Expected '{'application': 'MCX_Client', 'application_ref': 1000, 'result_code': 0}' on topic 'hmi/app_connect_response' but no such message received

Dec  1 10:25:39 5c02662b3add user.debug HMI[12]: [7fdcf1d3b740]on_log16 : Client HMI received PUBLISH (d0, q0, r0, m0, 'hmi/app_connect', ... (54 bytes))
Dec  1 10:25:39 5c02662b3add user.debug HMI[12]: [7fdcf1d3b740]virtual void MosquittoClient::on_message(const mosquitto_message*)
Dec  1 10:25:39 5c02662b3add user.debug HMI[12]: [7fdcf1d3b740]AppRequestHandler::handleAppConnect
Dec  1 10:25:39 5c02662b3add user.debug HMI[12]: [7fdcf1d3b740]HMIConfiguration::Instance
Dec  1 10:25:39 5c02662b3add user.debug HMI[12]: [7fdcf1d3b740]HMIConfiguration::validate_json_against_schema - ../json_schemas/HMI/target_app_connect_schema.json
Dec  1 10:25:39 5c02662b3add user.debug HMI[12]: [7fdcf1d3b740]AppRequestHandler::handleAppConnect data validated against schema
Dec  1 10:25:39 5c02662b3add user.debug HMI[12]: [7fdcf1d3b740]AppRequestHandler::isAppAlreadyConnected
Dec  1 10:25:39 5c02662b3add user.debug HMI[12]: [7fdcf1d3b740]AppRequestHandler::handleAppConnect - new app: MCX_Client
Dec  1 10:25:39 5c02662b3add user.debug HMI[12]: [7fdcf1d3b740]HMIConfiguration::Instance
Dec  1 10:25:39 5c02662b3add user.debug HMI[12]: [7fdcf1d3b740]HMIConfiguration::getPriorityForApp
Dec  1 10:25:39 5c02662b3add user.debug HMI[12]: [7fdcf1d3b740]HMIConfiguration::getPriorityForApp - found App 'MCX_Client' - priority = 2
Dec  1 10:25:39 5c02662b3add user.debug HMI[12]: [7fdcf1d3b740]AppRequestHandler::handleKeyRegistration
Dec  1 10:25:39 5c02662b3add user.debug HMI[12]: [7fdcf1d3b740]AppRequestHandler::sendAppConnectResponse 0
Dec  1 10:25:39 5c02662b3add user.debug HMI[12]: [7fdcf1d3b740]publish_to_topic (topic = 'hmi/app_connect_response', message = '{"application":"MCX_Client","application_ref":1000,"result_code":0}', qos = '0', retain = '0')

This happens for every stage in the test.

[michaelboulton]

I'm assuming that tavern does not report any messages on the topic you're listening to, not even one with an incorrect payload?