Functional testing with artillery-plugin-expect does not guarantee completeness

[LesterTheTester]

I am trying to use artillery-plugin-expect to also run my artillery tests as functional tests. However, the environment configuration shown to do this on the npm module page doesn't take into account the possiblity that one config file has multiple named scenarios:

config:
  environments:
    local:
      target: 'http://localhost:3000/'
      phases:
        - duration: 1
          arrivalRate: 1
      plugins:
        expect: {}
    staging:
      target: 'https://staging.myserver.com/'
      phases:
        - duration: 120
          arrivalRate: 10
  ensure:
    maxErrorRate: 1
scenarios:
  - name: Endpoint 1
    flow:
      - get:
          url: "/endpoint1"
          expect:
            - statusCode: 200
  - name: Endpoint 2
    flow:
      - get:
          url: "/endpoint2"
          expect:
            - statusCode: 200

It seems providing duration: 1 and arrivalRate: 1 means artillery will only ever execute 1 of my scenarios. Furthermore, due to the random selection, even if I change to arrivalRate: 2, I may just get two of the same scenario.

For proper functional testing, we need to validate ALL our scenarios in each run. Am I missing something here?

[hassy]

The way to go right now is to keep your scenarios in separate files, extract config to its own file too, and use the --config option to run your tests with common configuration. Your folder structure would look like this:

tests/
  - scenarios/
      - product-search.yaml
      - get-offers.yaml
      - checkout.yaml
  - config/
      - common.yaml

And you'd run your test with:

artillery run --config config/common.yaml scenarios/checkout.yaml

We're working on a way to run a number of given scenarios exactly once for this use-case for a future release of Artillery.

[LesterTheTester]

So it seems for now that I have to specify each one of my scenarios individually, artillery run --config src/e2e/config/common.yaml -e local src/e2e/scenarios/* results in just the first scenario being run. Is that correct?

[hassy]

Yes, that's correct. Artillery's run command does not guarantee that all scenarios given to it will be executed exactly once right now.

[vegerot]

@hassy how about now?

[jmrivas86]

Hi @hassy Is there news about this? thanks!