Newman skips request from folder and also does not run in order

[21anujashah]

Postman version is 7.3.4. I export collection in the JSON file. I run that collection from using Newman. I found that the requests are not running in order. I have one request to set the environment variable and the next request to take the environment and run it. But Newman runs the second request first, it does not have any environment. One thing, I found that my collection has more than 8 folders and each folder has subfolders. Each folder has many requests. I found that few requests are not run by Newman. Newman just skips the few requests and does not run in order. I do not write any code to skip request(i.e setNextRequest)

[jetzerb]

I wonder if this is related to #1851? In that issue, it is noted that specifying --folder only works for toplevel folders, and not nested paths such as --folder "path/to/subfolder" to run just the requests under path/to/subfolder.

Perhaps in your case, newman is confused by the multi-level folders. Have you tried re-arranging your collection to have just toplevel folders to see whether the behavior changes?

[codenirvana]

Can you share a scaled-down collection which we can use to reproduce this issue?

[ShahOdin]

I second the not-running-in-order issue. Here is my Json:

{"item":[{"item":[{"name":"getBar","event":[{"listen":"test","script":{"id":"b31d4bce-9611-4bd9-9cc5-907f393cf122","exec":["const bar = \"BAR_VALUE\";","pm.collectionVariables.set(\"bar\", bar);","console.log(\"bar is:\" + bar)"],"type":"text/javascript"}}],"request":{"method":"GET","url":{"raw":"https://postman-echo.com/delay/{{delayLength}}","protocol":"https","host":["postman-echo","com"],"path":["delay","{{delayLength}}"]}}}],"name":"Bar"},{"item":[{"name":"getFoo","event":[{"listen":"test","script":{"id":"b31d4bce-9611-4bd9-9cc5-907f393cf122","exec":["const foo = \"FOO_VALUE\";","pm.collectionVariables.set(\"foo\", foo);","console.log(\"foo is:\" +foo)"],"type":"text/javascript"}}],"request":{"method":"GET","url":{"raw":"https://postman-echo.com/delay/{{delayLength}}","protocol":"https","host":["postman-echo","com"],"path":["delay","{{delayLength}}"]}}}],"name":"Foo"},{"item":[{"name":"getFooBar","event":[{"listen":"test","script":{"id":"b31d4bce-9611-4bd9-9cc5-907f393cf122","exec":["const foo = pm.collectionVariables.get(\"foo\");","const bar = pm.collectionVariables.get(\"bar\");","const foobar = foo + \"*\" + bar;","pm.collectionVariables.set(\"fooBar\", foobar);","console.log(\"foobar is:\" + foobar)"],"type":"text/javascript"}}],"request":{"method":"GET","url":{"raw":"https://postman-echo.com/delay/{{delayLength}}","protocol":"https","host":["postman-echo","com"],"path":["delay","{{delayLength}}"]}}}],"name":"FooBar"}],"event":[],"variable":[],"info":{"_postman_id":"1f4443f7-dadb-4d1b-94f7-6b95fafc8345","name":"composed","schema":"https://schema.getpostman.com/json/collection/v2.1.0/collection.json"}}

I ran the following:

  newman run composed.postman_collection.json \
  -e envs.postman_environment.json \
  --folder Foo \
  --folder Bar \
  --folder FooBar \
  /

in the hope that they run in order. They don't.

[inamiy]

Related: https://github.com/postmanlabs/newman/issues/359

Same issue here. I don't have any way of running specific APIs in order.

newman run collection.json --folder api1 --folder api2 --folder api3

# Executes in different order...

(Pls note that --folder some_specific_api barely works regardless of the option name. )

[ShahOdin]

Can you share a scaled-down collection which we can use to reproduce this issue?

Here is a minimal collection with two requests:

• Foo
• Bar

in order.

{
	"info": {
		"_postman_id": "a2bb8ba2-51c1-4d94-b52a-c63ad2c63ed2",
		"name": "FooBar",
		"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
	},
	"item": [
		{
			"name": "Foo",
			"item": [
				{
					"name": "delay",
					"event": [
						{
							"listen": "test",
							"script": {
								"id": "a991f317-cfcb-47e6-8412-cafb03696b7b",
								"exec": [
									"console.log(\"foo\")"
								],
								"type": "text/javascript"
							}
						}
					],
					"request": {
						"method": "GET",
						"header": [],
						"url": {
							"raw": "https://postman-echo.com/delay/0",
							"protocol": "https",
							"host": [
								"postman-echo",
								"com"
							],
							"path": [
								"delay",
								"0"
							]
						}
					},
					"response": []
				}
			],
			"protocolProfileBehavior": {}
		},
		{
			"name": "Bar",
			"item": [
				{
					"name": "delay",
					"event": [
						{
							"listen": "test",
							"script": {
								"id": "370bf650-4a8b-4e35-9b55-769db0c327a9",
								"exec": [
									"console.log(\"bar\")"
								],
								"type": "text/javascript"
							}
						}
					],
					"request": {
						"method": "GET",
						"header": [],
						"url": {
							"raw": "https://postman-echo.com/delay/0",
							"protocol": "https",
							"host": [
								"postman-echo",
								"com"
							],
							"path": [
								"delay",
								"0"
							]
						}
					},
					"response": []
				}
			],
			"protocolProfileBehavior": {}
		}
	],
	"protocolProfileBehavior": {}
}

Running:

  newman run col.json \
  -e envs.postman_environment.json \
  --folder Foo \
  --folder Bar \
  --folder FooBar \
  /

behaves as expected: ie, Foo runs first, followed by Bar

whereas:

  newman run col.json \
  -e envs.postman_environment.json \
  --folder Bar \
  --folder Foo \
  --folder FooBar \
  /

still runs Foo first. one would expect it to run Bar first.

[adityaofficial10]

@codenirvana @shamasis I think the incorrect order of results is related to postman-runtime. Can I try to come up with a solution?

[adityaofficial10]

@shamasis I think of submitting a patch for this to postman-runtime.

[adityaofficial10]

@shamasis I have made the changes and opened the PR in postman-runtime. Kindly check.

[shamasis]

Also, if we do this, we need to put this behaviour behind an option so that this does not become a breaking change.

Ideally, what you see now (current newman) is expected behaviour. The collection determines the order and the folder parameter is merely a filter.

However, knowing that Postman Runner has opened a new feature to "reorder" items in collection in any way we see fit, we can proceed with this - but with an option.

[adityaofficial10]

Yeah actually @shamasis , you are right. How about having an option --order-folders for this purpose?

[shamasis]

Order folders will prevent us from reusing the same option for request filter when we build it next.

We should call this something that reflects that switching it on will respect the order of options over the predefined order.

[adityaofficial10]

@shamasis Okay, then what about orderStrict?? It reflects that the ordering is strict with respect to folders.

[shamasis]

This name is very close to what we want. But let's get more opinions.

@codenirvana @saswatds what do you think?

[adityaofficial10]

@codenirvana @shamasis I am submitting a patch for this. This will depend on the proposed change in the postman-runtime, for which the patch has already been submitted. I have informed GSoC mentors about my patches.