normandy icon indicating copy to clipboard operation
normandy copied to clipboard

New contract tests don't pass against stage or prod servers

Open mythmon opened this issue 6 years ago • 2 comments
trafficstars

The new contract tests work in CI (sometimes, see #1826), but don't pass against prod and stage. Looking at the problems in the test failures, it is definitely a shortcoming of the tests, not that the servers are currently broken.

This will prevent deploying the service, and is causing trouble with local development for me. I also suspect this is related to recent CI failures.

I'm going to revert the tests to their prior state, since this is now blocking our development. @chartjes, can you update the tests so that they are more reliable, and pass against our real deployments? After that we can re-land these tests.

mythmon avatar Mar 29 '19 21:03 mythmon

I am busy doing Android performance work so feel free to mark the tests as skipped — I will look at them after the all-hands in June

chartjes avatar Mar 29 '19 22:03 chartjes

Also I made this tests by taking some sample API calls and running them through a JSON schema generator. Some of the flakiness is likely due to me not knowing what fields are mandatory or optional.

chartjes avatar Mar 29 '19 22:03 chartjes