passivsystems
Testing of REST API methods using cross product of Mandatory, Optional and Unexpected parameters
Please treat as feature request:
As discussed, from a testing perspective there are 3 categories of http request parameter. Mandatory parameters, optionals and unexpected.
A typical manual QA approach to testing this would be as follows:
Generally as a minimum (assuming heavy time pressures for release):
Positive tests expecting a positive outcome 200/201 resp code etc.
Request attempt with all mandatory parameters provided using data values that are valid for the application logic. Request attempt with each optional parameter either individually or with all optionals.
Negative test expecting 404 resp code etc
Request attempt individually omitting each mandatory parameter
Should be done to provide a more thorough test coverage:
In addition of the above,
Positive tests
Request attempts with a selected range of optional values that match specific application use cases.
Negative tests
Request attempt with all possible permutations of mandatory parameter. Request attempt with mandatory values present but not valid for the application logic.
Test that are often not done especially after initial release (regression testing)
Positive
Full coverage of all permutations of optional parameters.
Negative
Full coverage of mandatory parameters omitted. Request attempts with unexpected parameters. Request attempts of cross product with mandatory (missing) options.
As you can see there are generally holes in test coverage on manual testing especially if the parameter list is quite long. It should be possible to automatically allow the creation of a test list from the API method signature that follows the general rules.
- Requests with all mandatory and any optional parameter will have a positive outcome
- Requests with any mandatory parameters missing will have a negative outcome, regardless of optional parameter presence.
- Requests with unexpected parameters should be either ignored or cause a negative result depending on organisation policy.
There would be occasions where the use of all mandatory parameters and some optionals would return a negative result based on application use of the data in the parameters so presumably would need some form of mechanism to handle departure from the rules.
Thanks for this. It seems almost like there could be two topics... the first of which is covering the different types of parameters, the second being a way to configure how rigorous the automated integration tests should be.
I am refactoring the test functionality at the moment, so I will attempt to roll some of these ideas in. I will let you know when a prototype is ready.
Required and optional parameter encoding capability has been built in. We do not currently iterate over each input and provide a range of values.
We do not yet handle nonsense parameters, this is on the roadmap.
branch integ_test_missing_required now supports level 3 integ testing: will test urls with a missing required form/query parameter - expecting a 400.
This is in addition to level 1 - required form/query params only, level 2 - optional form/query params.
This should also cover missing headers (with consideration that it may cause a different 4xx status - e.g. 401 for missing auth header).
We should also check invalid payloads - e.g. missing values, which are defined as required in schema (xml or json). This requires a lot of sophistication in parsing schemas and generating payloads accordingly, which we do not have at the moment.
level 3 integ test will also test invalid json where one of the required properties is missing. Currently only being applied for top level properties.
In addition, added the ability to generate json from a (simplistic) json schema. We should think about when this should be activated. For example, level 4 could switch on generated json, instead of using example/body json.