[🐛 Bug]: Cucumber profiles are not being loaded with WebDriverIO v7 and Cucumber 7

[hammzj]

Have you read the Contributing Guidelines on issues?

• [X] I have read the Contributing Guidelines on issues.

WebdriverIO Version

7.16.5

Node.js Version

12.20.1

Mode

WDIO Testrunner

Which capabilities are you using?

{
  "hostname": "localhost",
  "port": 4444,
  "path": "/",
  "user": "webdriverio",
  "key": "xxxxxxxxxxxxxxxx-xxxxxx-xxxxx-xxxxxxxxx",
  "region": "us",
  "specs": [
    "main/features/**"
  ],
  "suites": {
    "android": [
      "main/features/android/**/*.feature"
    ],
    "ios": [
      "main/features/ios/**/*.feature"
    ],
    "localizations_android": [
      "main/features/android/localizations/**/*.feature"
    ],
    "localizations_ios": [
      "main/features/ios/localizations/**/*.feature"
    ]
  },
  "exclude": [],
  "logLevel": "info",
  "outputDir": "log",
  "baseUrl": "http://localhost:8080",
  "framework": "cucumber",
  "cucumberOpts": {
    "timeout": 30000,
    "profile": [
      "dry-run"
    ],
    "tagExpression": "@android and @run and not @skip and not @wip and not @blocked"
  },
  "reporters": [
    "spec"
  ],
  "runner": "local",
  "specFileRetries": 0,
  "specFileRetriesDelay": 2,
  "specFileRetriesDeferred": false,
  "sync": true,
  "deprecationWarnings": true,
  "bail": 0,
  "waitforTimeout": 10000,
  "connectionRetryTimeout": 90000,
  "connectionRetryCount": 3,
  "services": [
    [
      "appium",
      {
        "args": {
          "relaxedSecurity": true,
          "address": "localhost",
          "basePath": "/wd/hub",
          "port": 4723
        },
        "command": "appium"
      }
    ]
  ],
  "capabilities": [
    {
      "appium:locale": "US",
      "appium:language": "en",
      "appium:deviceName": "pixel_3a",
      "appium:avd": "Pixel_3a_API_30",
      "appium:platformVersion": "11.0",
      "appium:app": "apps/android/my-app.apk",
      "appium:printPageSourceOnFindFailure": true,
      "excludeDriverLogs": [
        "bugreport",
        "performance"
      ],
      "platformName": "Android",
      "maxInstances": 1,
      "appium:orientation": "PORTRAIT",
      "appium:automationName": "UiAutomator2",
      "appium:noReset": false,
      "appium:fullReset": true,
      "appium:allowTestPackages": true,
      "appium:newCommandTimeout": 240,
      "goog:chromeOptions": {
        "args": [
          "no-first-run",
          "disable-fre",
          "no-default-browser-check"
        ]
      }
    }
  ]
}

What happened?

It looks like using a Cucumber profile does not load in correctly when specified in the capabilities. No matter what, it is ultimately skipped, and the Cucumber options must be directly specified with cucumberOpts, rendering the cucumberOpts.profile option as completely irrelevant.

I have a cucumber.js file defined at the root of my project directory as defined by here but even that does not do anything for me, even when I require the profiles before I finish loading my capabilities.

With the expanded capabilities above, I've tried to set a profile in two ways:

1. Specifying the profile I want to use directly, ie.

const caps = {
  cucumberOpts: {
    profile: ['dry-run'],
    timeout: 30000
  }
}

2. By expanding the profile string completely within the profile option:

const profiles = require('../../cucumber.js')
const caps = {
  cucumberOpts: {
    //--require '/Users/zachhamm/Documents/GitHub/automation-suite/main/support/constants.js' --require '/Users/zachhamm/Documents/GitHub/automation-suite/main/step_definitions/**/*.js' --require '/Users/zachhamm/Documents/GitHub/automation-suite/main/support/hooks/*.hooks.js' --tags '@android and @run and not @skip and not @wip and not @blocked' --format html:./cucumber-report.html --dry-run
    profile: [profiles['dry-run']],
    timeout: 30000
  }
}

Notice that I am not requiring any steps in the main cucumberOpts, but instead, requiring them through the profile. When I do that, it will fail because it can't find any of the steps to require.

If I select the dry-run profile using the cucumber-js command line option, it will load those required steps before running the feature and it works:

 cucumber-js main/features/android/zachtest.feature -p dry-run 
------------------------

3 scenarios (3 skipped)
9 steps (9 skipped)
0m00.036s (executing steps: 0m00.000s)

I've also tried where I've required the files in the cucumberOpts.require and then tried with the dry-run profile, hoping for a dry run, but alas, it still would not execute it as a dry run.

My conclusion is ultimately that the cucumberOpts.profile is not being loaded.

What is your expected behavior?

If loading a Cucumber profile in the cucumber.js file into the cucumberOpts.profile array, then the capabilities shall merge those options to the Cucumber framework as it does with the cucumberOpts object.

How to reproduce the bug.

1. Make sure your step definitions and hooks exist in a directory structure like such:

` cucumber.js
* apps
  * android
    ` my-app.apk 
* main
    * config
      ` environment.js
    * features
    * support
      * step_definitions
        ` *.steps.js 
      * hooks
        ` *.hooks.js  

2. Create a profile in the cucumber.js file as such:

// cucumber.js
const path = require('path');
const dryRun = [
`--require '${path.join(__dirname, 'main/support/step_definitions/**/*.js'}'`,
`--require '${path.join(__dirname, 'main/support/hooks/**/*.js'}'`
`--format progress-bar`
`--dry-run`
].concat(' ');

modules.export = {
'dry-run': dryRun 
}

3. Use the capabilities (replacing your device and feature paths as needed): main/config/environment.js

{
  "hostname": "localhost",
  "port": 4444,
  "path": "/",
  "user": "webdriverio",
  "key": "xxxxxxxxxxxxxxxx-xxxxxx-xxxxx-xxxxxxxxx",
  "region": "us",
  "specs": [
    "main/features/**"
  ],
  "logLevel": "info",
  "outputDir": "log",
  "baseUrl": "http://localhost:8080",
  "framework": "cucumber",
  "cucumberOpts": {
    "timeout": 30000,
    "profile": [
      "dry-run"
    ],
    "tagExpression": "@android and @run and not @skip and not @wip and not @blocked"
  },
  "reporters": [
    "spec"
  ],
  "runner": "local",
  "specFileRetries": 0,
  "specFileRetriesDelay": 2,
  "specFileRetriesDeferred": false,
  "sync": true,
  "bail": 0,
  "waitforTimeout": 10000,
  "connectionRetryTimeout": 90000,
  "connectionRetryCount": 3,
  "services": [
    [
      "appium",
      {
        "args": {
          "relaxedSecurity": true,
          "address": "localhost",
          "basePath": "/wd/hub",
          "port": 4723
        },
        "command": "appium"
      }
    ]
  ],
  "capabilities": [
    {
      "appium:deviceName": "pixel_3a",
      "appium:avd": "Pixel_3a_API_30",
      "appium:platformVersion": "11.0",
      "appium:app": "apps/android/my-app.apk",
      "platformName": "Android",
      "maxInstances": 1,
      "appium:automationName": "UiAutomator2",
      "appium:allowTestPackages": true,
      "appium:newCommandTimeout": 240,
    }
  ]
}

4. Tag a feature to runwith @android @run
5. Run the client:

node wdio main/config/environment.js 

Relevant log output

"spec" Reporter:
------------------------------------------------------------------
[emulator-5554 Android 11 #0-0] Running: emulator-5554 on Android 11 executing apps/android/my-app.apk
[emulator-5554 Android 11 #0-0] Session ID: 5724913e-9201-4739-9e5e-bf4ff5023607
[emulator-5554 Android 11 #0-0]
[emulator-5554 Android 11 #0-0] » /main/features/android/zachtest.feature
[emulator-5554 Android 11 #0-0] Android | TEST
[emulator-5554 Android 11 #0-0] Making sure stuff runs
[emulator-5554 Android 11 #0-0]    ✖ Given the user is on the login screen
[emulator-5554 Android 11 #0-0]    ✖ When the user taps the forgotPasswordLink element
[emulator-5554 Android 11 #0-0]    ✖ Then the user is on the forgotPassword screen
[emulator-5554 Android 11 #0-0]    ✖ And the following elements are displayed:
[emulator-5554 Android 11 #0-0]      │ cancelButton │
[emulator-5554 Android 11 #0-0]
[emulator-5554 Android 11 #0-0] 4 failing (1.3s)
[emulator-5554 Android 11 #0-0]
[emulator-5554 Android 11 #0-0] 1) Making sure stuff runs Given the user is on the login screen
[emulator-5554 Android 11 #0-0] Step "the user is on the login screen" is not defined. You can ignore this error by setting cucumberOpts.ignoreUndefinedDefinitions as true.
[emulator-5554 Android 11 #0-0] Step "the user is on the login screen" is not defined. You can ignore this error by setting cucumberOpts.ignoreUndefinedDefinitions as true.
[emulator-5554 Android 11 #0-0]         at Feature(/Users/zachhamm/Documents/GitHub/automation-suite/main/features/android/zachtest.feature):1:1
[emulator-5554 Android 11 #0-0]         at Scenario(Making sure stuff runs):35:3
[emulator-5554 Android 11 #0-0]         at Step(the user is on the login screen):38:5
[emulator-5554 Android 11 #0-0]
[emulator-5554 Android 11 #0-0]
[emulator-5554 Android 11 #0-0] 2) Making sure stuff runs When the user taps the forgotPasswordLink element
[emulator-5554 Android 11 #0-0] Step "the user taps the forgotPasswordLink element" is not defined. You can ignore this error by setting cucumberOpts.ignoreUndefinedDefinitions as true.
[emulator-5554 Android 11 #0-0] Step "the user taps the forgotPasswordLink element" is not defined. You can ignore this error by setting cucumberOpts.ignoreUndefinedDefinitions as true.
[emulator-5554 Android 11 #0-0]         at Feature(/Users/zachhamm/Documents/GitHub/automation-suite/main/features/android/zachtest.feature):1:1
[emulator-5554 Android 11 #0-0]         at Scenario(Making sure stuff runs):35:3
[emulator-5554 Android 11 #0-0]         at Step(the user taps the forgotPasswordLink element):39:5
[emulator-5554 Android 11 #0-0]
[emulator-5554 Android 11 #0-0]
[emulator-5554 Android 11 #0-0] 3) Making sure stuff runs Then the user is on the forgotPassword screen
[emulator-5554 Android 11 #0-0] Step "the user is on the forgotPassword screen" is not defined. You can ignore this error by setting cucumberOpts.ignoreUndefinedDefinitions as true.
[emulator-5554 Android 11 #0-0] Step "the user is on the forgotPassword screen" is not defined. You can ignore this error by setting cucumberOpts.ignoreUndefinedDefinitions as true.
[emulator-5554 Android 11 #0-0]         at Feature(/Users/zachhamm/Documents/GitHub/automation-suite/main/features/android/zachtest.feature):1:1
[emulator-5554 Android 11 #0-0]         at Scenario(Making sure stuff runs):35:3
[emulator-5554 Android 11 #0-0]         at Step(the user is on the forgotPassword screen):40:5
[emulator-5554 Android 11 #0-0]
[emulator-5554 Android 11 #0-0]
[emulator-5554 Android 11 #0-0] 4) Making sure stuff runs And the following elements are displayed:
[emulator-5554 Android 11 #0-0] Step "the following elements are displayed:" is not defined. You can ignore this error by setting cucumberOpts.ignoreUndefinedDefinitions as true.
[emulator-5554 Android 11 #0-0] Step "the following elements are displayed:" is not defined. You can ignore this error by setting cucumberOpts.ignoreUndefinedDefinitions as true.
[emulator-5554 Android 11 #0-0]         at Feature(/Users/zachhamm/Documents/GitHub/automation-suite/main/features/android/zachtest.feature):1:1
[emulator-5554 Android 11 #0-0]         at Scenario(Making sure stuff runs):35:3
[emulator-5554 Android 11 #0-0]         at Step(the following elements are displayed:):41:5
[emulator-5554 Android 11 #0-0]


Spec Files:      0 passed, 1 failed, 1 total (100% completed) in 00:01:08

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• [X] I have searched the existing issues

[christian-bromann]

@hammzj thanks for the thorough written issue. This seems like a feature request to support cucumber profiles. I think it is a good idea! Please get involved and help us make this reality. Please take a look into our contribution guidelines and let us know if you have any questions. Cheers!

[hammzj]

@christian-bromann, definitely. However, I believe it is a bug because WebDriverIO's Testrunner Configuration does have an option for Cucumber profiles in both v6 and v7.

   // If you are using Cucumber you need to specify where your step definitions are located.
    // See also: https://github.com/webdriverio/webdriverio/tree/main/packages/wdio-cucumber-framework#cucumberopts-options
    cucumberOpts: {
        require: [],        // <string[]> (file/dir) require files before executing features
        backtrace: false,   // <boolean> show full backtrace for errors
        compiler: [],       // <string[]> ("extension:module") require files with the given EXTENSION after requiring MODULE (repeatable)
        dryRun: false,      // <boolean> invoke formatters without executing steps
        failFast: false,    // <boolean> abort the run on first failure
        format: ['pretty'], // <string[]> (type[:path]) specify the output format, optionally supply PATH to redirect formatter output (repeatable)
        snippets: true,     // <boolean> hide step definition snippets for pending steps
        source: true,       // <boolean> hide source URIs
        profile: [],        // <string[]> (name) specify the profile to use
        strict: false,      // <boolean> fail if there are any undefined or pending steps
        tagExpression: '',  // <string> (expression) only execute the features or scenarios with tags matching the expression
        timeout: 20000,     // <number> timeout for step definitions
        ignoreUndefinedDefinitions: false, // <boolean> Enable this config to treat undefined definitions as warnings.
        scenarioLevelReporter: false // Enable this to make webdriver.io behave as if scenarios and not steps were the tests.
    },

It is also in the wdio-cucumber-framework package on GitHub.

If I get free time I'll take a look into it more, but I wanted to raise it for awareness. Thanks.

[christian-bromann]

Yeah, we have this option but was never actually implemented.

If I get free time I'll take a look into it more

Awesome, let me know if I can support in any way.

[hammzj]

So @christian-bromann, ultimately, this property probably should be removed entirely unless we allow it only by command line. profile just allows us to merge additional options into the cucumberOpts that aren't set as the default in a wdio.conf.js file. Necessarily, using a profile will overwrite or add new properties onto what exists within the environment file originally and therefore the main environment file may be inaccurate to read.

Because of this, it wouldn't make sense to use them for both

1. a single main environment file, where profiles or
2. mutliple wdio.conf.js environment files.

In both of these cases, a Cucumber profile should only take effect on the command line because whatever would get put in the separate Cucumber profiles list file would take immediate precedence. Then, there could be wildly different options set than what is in the main cucumberOpts, turning it into maintenance hell, due to the preferred options existing in a separate file (profiles.js) than in the wdio.conf.js file.

Finally, we would have to have a type for each option that could be set, and make sure that Cucumber profile strings are converted to those internal types before

As a workaround, what I've done is use a new argument that merges a separate object of cucumberOpts from other files onto my main environment file. That way, I can still use a "profile" of sorts, but I only enable it via a command line option.

Please let me know if this makes sense. I'm on the fence now about adding it in instead of removing it, unless it is is preferred as a command line option.

[platon023]

Good afternoon, a very useful function would be if it would be possible to add parameters to WebdriverIO from cucumber.js What ways can this be done?

[christian-bromann]

@tamil777selvan do you know if profiles are now supported with the new API interface we use?

[tamil777selvan]

@christian-bromann At present, profiles are inactive because the values in profiles are consistently overridden by cucumberOpts.

[tamil777selvan]

Hi @christian-bromann This PR enables support of cucumber profiles with limited options, as cucumberOpts takes precedence.

[christian-bromann]

Thanks @tamil777selvan , we can go ahead and close this then.