External config with environments receives error

[efabisze]

Version info:

Artillery Core: 2.0.0-12

Running this command:

Running a script with config with environments in separate file and script run gets error : 
artillery run -e production --config *_filePath_*/config.yaml *_filePath_*/TimeSlots.yaml

I expected to see this happen:

the load test run

Instead, this happened:

WARNING: environment production is set but is not defined in the script

Attempted to debug and adding config, to same file , allowed the script to run. It seems to be when the config is in a separate file I used the artillery documented environments config to check if its my custom config and received same error Files being used:

<relevant yaml/js/csv go here>

[rmjwilbur]

I'm experiencing this same issue. My test still runs. DEBUG=http doesn't provide any info.

From config:

environments:
    dev:
      phases:
        - duration: 1
          arrivalCount: 1
          maxVusers: 1
          name: Phase 1

From scenarios:

scenarios:
  - flow:
      - log: "Environment: {{ $environment }}"

Output:

    output: WARNING: environment dev is set but is not defined in the script
    Test run id: tdb45_d85t9qygkhjbgmc9qtcbzrzr6fhtc_393x
    Phase started: Phase 1 (index: 0, duration: 1s) 11:57:34(-0700)
    
    Phase completed: Phase 1 (index: 0, duration: 1s) 11:57:35(-0700)
    
    Environment: dev

Note that the Phase notices include the name defined in the dev environment section

[eliezercazarestravelport]

Just found this is going on for me too. I think it was working fine a few weeks ago, not 100% sure.

One thing to note: if you remove the environment from the config file, or rename it to something different so your execution does not have a valid environment you get TWO warnings about "environment not defined", instead of one as in your case scenario.

That makes me think Artillery is now processing both files independently.

[eliezercazares]

I'm experiencing this same issue. My test still runs. DEBUG=http doesn't provide any info.

From config:

environments:
    dev:
      phases:
        - duration: 1
          arrivalCount: 1
          maxVusers: 1
          name: Phase 1

From scenarios:

scenarios:
  - flow:
      - log: "Environment: {{ $environment }}"

Output:

    output: WARNING: environment dev is set but is not defined in the script
    Test run id: tdb45_d85t9qygkhjbgmc9qtcbzrzr6fhtc_393x
    Phase started: Phase 1 (index: 0, duration: 1s) 11:57:34(-0700)
    
    Phase completed: Phase 1 (index: 0, duration: 1s) 11:57:35(-0700)
    
    Environment: dev

Note that the Phase notices include the name defined in the dev environment section

Your test still runs fine UNLESS it's a "Functional" test. This scenario: having an external config with environments is not working for Functional environments. I had to add a redundant config to all my tests because I will demo to a large company tomorrow in an attempt to have them use something better than a paid and very bad option.

It is sad that "Functional" tests are not working with External Config Files in latest Artillery. Hopefully an admin can take a look at this post soon.

[eliezercazares]

@hassy sorry I tag you Hassy, but you're listed as the first contributor so I went ahead...

Can you take a look at my latest comment? In short: "Functional" tests are not working with External Config Files in latest Artillery version.

We get the error that the OP states. (I cannot copy paste any info of my employer cause I am working an FTE). But it is the same error, and I tried to summarize it as short as possible for you above.

Thanks a lot!

[hassy]

@eliezercazares will take a look now

[hassy]

The following seems to work for me as expected, unless I've misunderstood your post @eliezercazares:

# config.yml
config:
  target: "http://asciiart.artillery.io:8080"
  environments:
    dev:
      phases:
        - duration: 1
          arrivalCount: 1
          maxVusers: 1
          name: Phase 1

# scenarios.yaml
scenarios:
  - flow:
      - log: "Environment: {{ $environment }}"

Running with:

artillery run --config config.yml scenarios.yml -e dev

Produces expected output: ("Environment: dev")

Test run id: tah59_9ztqbwwr5a8aha8qf53wq7pammyeg_hpdr
Phase started: Phase 1 (index: 0, duration: 1s) 19:09:41(+0100)

⠧ Environment: dev
Phase completed: Phase 1 (index: 0, duration: 1s) 19:09:42(+0100)

--------------------------------------
Metrics for period to: 19:09:50(+0100) (width: 0.007s)
--------------------------------------

vusers.completed: .............................................................. 1
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 6.6
  max: ......................................................................... 6.6
  median: ...................................................................... 6.6
  p95: ......................................................................... 6.6
  p99: ......................................................................... 6.6


All VUs finished. Total time: 2 seconds

--------------------------------
Summary report @ 19:09:45(+0100)
--------------------------------

vusers.completed: .............................................................. 1
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 6.6
  max: ......................................................................... 6.6
  median: ...................................................................... 6.6
  p95: ......................................................................... 6.6
  p99: ......................................................................... 6.6

[hassy]

@eliezercazares let me know if I should try a slightly different configuration that breaks for you. I'm running the latest version of Artillery above - v2.0.0-35.

[eliezercazares]

@eliezercazares let me know if I should try a slightly different configuration that breaks for you. I'm running the latest version of Artillery above - v2.0.0-35.

Perhaps I messed up with the wording... let me try to provide an example...

config.yml

config:
  http:
    timeout: 60
  plugins:
    json-include: {}
    metrics-by-endpoint: {}
  target: ""
  processor: ""
  environments:
    load:
      - name: "Load"
        phases:
          - duration: 10
             arrivalRate: 1
    functional:
      - name: "Functional"
        plugins:
          expect: {}

test.yml

scenarios:
  - name: ""
    flow:
      - post: 
        url: ""
        json: ""
       capture: 
         - json: ""
           as: myVar
       expect:
         - statusCode: 200 

Run with:

artillery run -e functional --config config.yml scenarios/test.yml

I really think you can use any test. It's when you have more than one Environment and external config file.

[hassy]

Thanks @eliezercazares. There's a syntax error in the config definition you shared. If you change the envoronments section to:

  environments:
    load:
      phases:
        - duration: 10
          arrivalRate: 1
    functional:
      plugins:
        expect: {}

Then it should work as expected. In the config you shared the value of load and functional is an array, but it needs to be an object.

[eliezercazares]

Thanks @eliezercazares. There's a syntax error in the config definition you shared. If you change the envoronments section to:

  environments:
    load:
      phases:
        - duration: 10
          arrivalRate: 1
    functional:
      plugins:
        expect: {}

Then it should work as expected. In the config you shared the value of load and functional is an array, but it needs to be an object.

I modified, still getting the warning... now it looks like this:

config:
  http:
     timeout: 60
  plugins:
    json-include: {}
    metrics-by-endpoint: {}
  target: ""
  processor: ""
  environments:
    load:
      phases:
        - duration: 10
           arrivalRate: 1
    functional:
      plugins:
        expect: {}

Right after entering "artillery run ......" command I get the:

WARNING: environment functional is set but is not defined in the script.

BUT after the changes you pointed out, I do see EXPECT getting executed and returning values, which means it is now working, but we still get the Warning.

[eliezercazares]

Thanks @eliezercazares. There's a syntax error in the config definition you shared. If you change the envoronments section to:

  environments:
    load:
      phases:
        - duration: 10
          arrivalRate: 1
    functional:
      plugins:
        expect: {}

Then it should work as expected. In the config you shared the value of load and functional is an array, but it needs to be an object.

Since Expect is working, I will be alright in my demo tomorrow... but (perhaps I messed up something else?) we still see the WARNING mentioned above. For now, I will focus on preparing my demo to our VP since we have been dealing with a BDD Eclipse based tool that plainly sucks. It crashes all the time, in the other hand Artillery it's so lightweight, practical, readable, so much potential.

THANKS A LOT HASSY! I am still available for the Warning being printed out in the CMD window.

[hassy]

Glad it's working @eliezercazares! Puzzled about why the warning is still showing though. 🤔 Could you confirm that you're running the latest version of Artillery? (with artillery version)

we have been dealing with a BDD Eclipse based tool that plainly sucks. It crashes all the time, in the other hand Artillery it's so lightweight, practical, readable, so much potential.

Thank you, very happy to hear that!

If you'd like to try Artillery Cloud dashboard (https://www.artillery.io/cloud) drop me a line on [email protected] - it provides much more powerful visualizations than the built-in artillery report command for load tests. We're in closed beta right now but happy to give you early access.

[eliezercazares]

Glad it's working @eliezercazares! Puzzled about why the warning is still showing though. 🤔 Could you confirm that you're running the latest version of Artillery? (with artillery version)

we have been dealing with a BDD Eclipse based tool that plainly sucks. It crashes all the time, in the other hand Artillery it's so lightweight, practical, readable, so much potential.

Thank you, very happy to hear that!

If you'd like to try Artillery Cloud dashboard (https://www.artillery.io/cloud) drop me a line on [email protected] - it provides much more powerful visualizations than the built-in artillery report command for load tests. We're in closed beta right now but happy to give you early access.

Hey thank you for all your help! I think the guys at the company would be interested in watching some of that. Not sure if I can pull it up for tomorrow's demo but I will make sure I can walk them through in a second meeting or a follow up video.

I was on v 2.0.0-27 but I am doing an update right now.

[eliezercazares]

Sorry for the delay got stuck with work, so it happened in the latest version yet!