Collection Runner and Newman: a request that got no response does not abort the collection, no way to do so programmatically

[jhardin-aptos]

I have a fairly large test collection that I'm working on right now. This morning I fired it off in Newman with HTML output against my dev environment and... it just sat there.

Disabling HTML output showed pretty clearly that the webservice wasn't responding - oops, forgot to start it up. D'oh! :(

So I've been digging around in the Postman docs to see how to catch this and do postman.setNextRequest(null); to abort the collection, and... there's no programmatic way to capture "no response" - and getting no response to a request (obviously) does not automatically abort the collection run.

This isn't so bad in Collection Runner because that's interactive and the user can abort it easily enough, but Newman is probably running the tests unattended.

To Reproduce

1. Open a multi-request test collection
2. Point it at a webservice that will not respond
3. Run it using Newman
4. The first request gets "no response" (e.g. ECONNREFUSED)
5. Subsequent requests still execute

Expected behavior Either:

1. The collection should abort (but I can see reasons to not do this automatically) or
2. Expose some way to control the collection behavior on a request failure

Newman does have the -bail failure option, but that's too broad - I do want to run multiple tests and verify their results and see all the failed test conditions; bailing on the first test failure isn't desirable.

Perhaps -bail needs a request_failure option too?

Perhaps (also?) there needs to be an option to run a javascript snippet if the request fails. This would be more flexible, but would be a more-intrusive change to both Newman and Postman...

App information: Newman 4.3.1 on Windows 10

[jhardin-aptos]

I attempted this to abort the collection promptly if the service is not responding:

1. Have the first request be a "warm the service" request with a simple lightweight request of the service
2. In that request's pre-request script do postman.setNextRequest(null);
3. In that request's tests script do postman.setNextRequest(2);

...but it doesn't work, the collection always terminates after the first request. Apparently only the first call to setNextRequest() is respected.

Can we please get some clear documentation of how setNextRequest() behaves? Details to clarify:

1. When does it take effect? Immediately, or at the end of the processing of the current request? (I had to figure this out by trial-and-error...)
2. The problem illustrated by the above: can you override a setNextRequest() with a subsequent one? Or is only the first one that is executed in a given request effective? (I had to figure this out by trial-and-error, too)

Note: answering those questions here in issue comment discussion is not documentation.

[shamasis]

Moved it to newman and would make part of release that tackles better execution control.

• allowing failed requests to still run test script
• option to bail on request error

[govind794]

I was able to break the flow using below code. Write it in the test script of first api, if api response returns success = false, it would break the flow. pm.test("API response", function() { var body = pm.response.json(); let identical = _.isEqual(body.success, true); if (identical === false) { postman.setNextRequest(null); } });

[spenceclark]

Is this issue still open?

I have a request very early on in my collection that checks to make sure a dependancy is up.

My test looks like this:

`// Check for success pm.test("Status code is 200", function () { pm.response.to.have.status(200); });

// Parse response var resp = pm.response.json();

pm.test("Maintenance mode = false", function () { pm.expect(resp.inMaintenanceMode).to.be.eql(false); });

if (resp.inMaintenanceMode === true) postman.setNextRequest(null);`

When I run via Postman app - it stops at that point if the flag is true. However the exact same pack run via Newman carries on to the next request.

[spenceclark]

Is this issue still open?

I have a request very early on in my collection that checks to make sure a dependancy is up.

My test looks like this:

`// Check for success pm.test("Status code is 200", function () { pm.response.to.have.status(200); });

// Parse response var resp = pm.response.json();

pm.test("Maintenance mode = false", function () { pm.expect(resp.inMaintenanceMode).to.be.eql(false); });

if (resp.inMaintenanceMode === true) postman.setNextRequest(null);`

When I run via Postman app - it stops at that point if the flag is true. However the exact same pack run via Newman carries on to the next request.

Actually no, this does seem to be working as expected in Newman when i tried again today; it must have been something I did wrong.