pact-net icon indicating copy to clipboard operation
pact-net copied to clipboard

ProviderVerifier is not failing the test when PublishResults part has failed downstream

Open xeroeugene opened this issue 3 years ago • 1 comments
trafficstars

BrokerSource (and UriSource in the future #392 ) have capability to publish verification results to the broker.

However underlying implementation might fail to publish the results (for many reasons), and that has no effect on the test results.

In other words your test would still be marked as passed and green, pact verification succeeded, but the results are not published anywhere and there is no way to know that, unless you are manually monitoring the logs.

This is especially important with webhook based scenarios, when the provider side builds are triggered on demand by the broker on pact contract changes. (no human would be involved to read the logs to understand that there was a problem)

Example from logs: [ERROR][pact_verifier] Publishing of verification results failed with an error when a read-only broker token was used to publish the results

Potential solution options:

  1. Make verification results publishing error propagation the default behavior, causing the provider verification test to fail
  2. Make verification results publishing error propagation an optional capability that can be switched on and off in the publishing options

xeroeugene avatar Jun 17 '22 04:06 xeroeugene

This is probably a defect in the FFI not reporting the failure back to PactNet. I'll have to check the FFI source.

adamrodger avatar Jun 18 '22 08:06 adamrodger