Invalid vector response "passes" on demo

[mmccarl]

I'm not sure that this would be classified as a bug or a feature, but I encountered a scenario using the Demo system that is misleading.

after generating test vectors, i submitted a response file that did not pass validation as indicated by the verdicts response. because this was on the demo system, a new vector response file was generated, submitted and passed as indicated by the verdicts.

because the test harness contained a bunch of debugging stuff, it was modified to remove the excess code. to be safe, the request file was re-run and actually produced an incorrect response file which was submitted to to the demo system. this time, however, the verdicts came back as 'passed'.

it would be a good idea to note somewhere that once passing verdicts are received from the demo system that the demo system will disregard subsequent submissions and return the passing verdicts. other options might be to actually process every submission and return the correct verdicts; or simply reject submissions after a passing submission was processed.

this was all confirmed under test session 255987

[livebe01]

This is interesting. I don't think the behavior that you're describing is what most people would expect. We'll take a look and look into the best way to resolve this.

[jbrock24]

@livebe01 Is there a way we can tackle this now without a session? Not sure what was done to trigger this response, if maybe @mmccarl can remember what was done and submit another response, I can look into it. If there's another way to get it without that, even better.

[mmccarl]

It appears that this issue has been fixed. using test session 336961, i submitted an erroneous response file and received a 'failed' verdict. next i submitted a correct response file and received a 'passed' verdict. then i sent a failed file and received the following message: "error": "Validation error(s) on JSON payload.", "context": [ "Vector set not in 'failed' or error status.", "Unable to submit results when vector set in 'passed' status."