bug: Skip functionality is unreliable in tests

[fokion]

Skip functionality is not reliable at the moment , created a small test scenario below just to validate my suspicions :

name: "Testing skip"
vars:
  features:
    - ACCOUNT
    - ACCOUNTS
    - USERS
  featuresAsMap:
      ACCOUNT: enabled
  foo: "bar"
testcases:
  - name: skip-as-we-dont-have-USER
    skip:
      - features MustContain "USER"
    steps:
      - type: exec
        script: exit 1
  - name: dont-skip-as-we-have-ACCOUNT-in-array
    skip:
      - features ShouldContain "ACCOUNT"
    steps:
      - type: exec
        name: Check-Global-Var
        script: echo "success"
        info:
          - "{{.result.systemout}}"
  - name : do-not-skip-as-its-an-array
    skip:
      - features ShouldBeArray
    steps:
      - type: exec
        name: Check-Global-Var
        script: echo "success"
        info:
          - "{{.result.systemout}}"
  - name: do-not-skip-as-its-a-map
    skip:
      - featuresAsMap ShouldBeMap
    steps:
      - type: exec
        name: Check-Global-Var
        script: echo "success"
        info:
          - "{{.result.systemout}}"
  - name: do-not-skip-as-its-a-map-with-key
    skip:
      - featuresAsMap ShouldContainKey "ACCOUNT"
    steps:
      - type: exec
        name: Check-Global-Var
        script: echo "success"
        info:
          - "{{.result.systemout}}"
  - name: do-not-skip-as-its-a-map-with-key-val
    skip:
      - featuresAsMap.ACCOUNT ShouldEqual "enabled"
    steps:
      - type: exec
        name: Check-Global-Var
        script: echo "success"
        info:
          - "{{.result.systemout}}"
  - name: dont-skip-val
    skip:
      - foo ShouldEqual "bar"
    steps:
      - type: exec
        name: Check-Global-Var
        script: echo "success"
        info:
          - "{{.result.systemout}}"
  - name: Check-Local
    steps:
      - type: exec
        script: |
          echo "{\"k\":[\"ACCOUNT\",\"USERS\"]}"
        vars:
          otherJson:
            from: result.systemoutjson.k
      - type: exec
        name: do-not-skip-as-local-json-variable-has-ACCOUNT
        skip:
          - otherJson MustContain ACCOUNT
        script: echo "success"
        info:
          - "{{.result.systemout}}"
  - name : Check-Local-2
    steps:
      - type: exec
        script: |
          echo "test"
        vars:
          other:
            from: features
      - type: exec
        name: do-not-skip-as-local-variable-has-ACCOUNT
        skip:
          - other MustContain ACCOUNT
        script: echo "success"
        info:
          - "{{.result.systemout}}"

Most cases are skipped even though they should not be.

The main issue is with how we check Arrays and Map items

[fokion]

name: "Testing skip"
testcases:
   - name : CHECK
     steps :
       - type: exec
         script: |
           echo "{\"k\":[\"ACCOUNT\",\"USERS\"]}"
         vars:
           blob:
             from: result.systemoutjson.k
           k:
             from : result.systemoutjson
   - name : OTHER-MUST-NOT-BE-SKIPPED
     skip :
       - CHECK.k MustNotBeNil
     steps:
       - type: exec
         script : echo "hello"
         info :
           - '{{.CHECK.k}}'
   - name : OTHER-WITHOUT-SKIP
     steps:
       - type: exec
         script : echo "hello"
         info :
           - '{{.CHECK.k}}'

This is not working as well

[yesnault]

ref https://github.com/ovh/venom/pull/685