ensure not throwing error

[lazyseals]

Setting I have a very basic configuration file with 3 ensure checks.

config:
  target: <omitted-target>
  phases:
    - duration: 20
      arrivalRate: 5
      rampTo: 10
      name: Sustained load
  ensure:
    p95: 100
    max: 200
    maxErrorRate: 1

scenarios:
  - name: "Visit Landing"
    flow:
      - get:
          url: "/"

I run artillery/2.0.0-10 win32-x64 node-v14.15.3.

What's going wrong? Running this test I get the following report summary.

--------------------------------
Summary report @ 12:48:14(+0100)
--------------------------------

vusers.created_by_name.Visit Landing: .......................................... 148
vusers.created.total: .......................................................... 148
vusers.completed: .............................................................. 148
vusers.session_length:
  min: ......................................................................... 197.4
  max: ......................................................................... 422.1
  median: ...................................................................... 247.2
  p95: ......................................................................... 320.6
  p99: ......................................................................... 347.3
http.request_rate: ............................................................. 8/sec
http.requests: ................................................................. 148
http.codes.200: ................................................................ 148
http.responses: ................................................................ 148
http.response_time:
  min: ......................................................................... 101
  max: ......................................................................... 274
  median: ...................................................................... 138.4
  p95: ......................................................................... 194.4
  p99: ......................................................................... 242.3
Log file: test.json

As you can see the ensure condition p95 = 194.4 > 100 = ensure.p95 is violated. Unfortunately, there is no error thrown stating that the respective ensure condition has failed.

What's expected? I would expect to get an error like ensure condition failed: ensure.p95 < 100

[jclusso]

Same issue here

[hassy]

hi folks 👋 got a fix for you!

1. We just updated the docs to mention that using ensure in Artillery v2 needs artillery-plugin-ensure to be installed. Sorry about that, there was no way for users to find that out.
2. We just released a new version of the ensure plugin which makes it a lot more powerful. You can still set simple threshold checks, but you can also create more complex conditions based on more than one metric.

Everything is in the docs here: https://www.artillery.io/docs/guides/guides/test-script-reference#ensure---slo-checks

[jclusso]

@hassy I still can't get it working... it just doesn't honor my simple ensure.

config:
  ensure:
    thresholds:
      # median of response time must be <250:
      - "http.response_time.median": 250

[hassy]

@jclusso oops, a case of documentation being updated before a new release and going out of sync 🤦‍♂️ Could you try again with the latest version of artillery and artillery-plugin-ensure please?

Example script to try with:

config:
  target: "https://www.artillery.io"
  plugins:
    ensure: {}
  phases:
    - duration: 10
      arrivalRate: 1
  ensure:
    thresholds:
      - http.response_time.p95: 1000
    conditions:
      - expression: http.response_time.p99 < 100 and vusers.created <= 10
        strict: false
      - expression: vusers.failed == 0
      - expression: http.codes.200 > 10
    p99: 1000
    maxErrorRate: 0
scenarios:
  - flow:
      - get:
          url: "/"
      - get:
          url: "/docs"
      - get:
          url: "/integrations"

[jclusso]

@hassy I have the following for testing and nothing changed. It still just continues to run.

config:
  plugins:
    ensure: {}
    expect: {}

  http:
    timeout: 360

  ensure:
    thresholds:
      - http.response_time.median: 1
    median: 1
    maxErrorRate: 0

  ...

Is this the latest version?

$ artillery -v

        ___         __  _ ____
  _____/   |  _____/ /_(_) / /__  _______  __ ___
 /____/ /| | / ___/ __/ / / / _ \/ ___/ / / /____/
/____/ ___ |/ /  / /_/ / / /  __/ /  / /_/ /____/
    /_/  |_/_/   \__/_/_/_/\___/_/   \__  /
                                    /____/


VERSION INFO:

Artillery Core: 2.0.0-11
Artillery Pro:  3.0.0-4

Node.js: v17.4.0
OS:      darwin

[jclusso]

@hassy any update?

[lazyseals]

Sorry for the late response. I can confirm that the issue is resolved with @hassy's response. Thank you!