[🐛 Bug]: Number of workers created not filtered by cucumber tags

[darnold24]

Have you read the Contributing Guidelines on issues?

• [X] I have read the Contributing Guidelines on issues.

WebdriverIO Version

7.16.16

Node.js Version

16.3.0

Mode

WDIO Testrunner

Which capabilities are you using?

No response

What happened?

When running yarn test:features in the root of the cucumber-boiler plate project (link provided in steps to reproduce), the number of workers created does not take cucumber tag filtering into account. In the file cucumber-boilerplate/wdio.conf.ts on line 176, if I replace tagExpression: 'not @Pending', with tagExpression: '@test', and then add the @test tag to 3 cucumber scenarios I see the following in the terminal.

╰─ yarn test:features
yarn run v1.22.10
$ wdio run wdio.BUILD.conf.ts

Execution of 30 workers started at 2022-04-19T21:06:27.792Z

This does not prevent the tests from starting, however in the case of our project, 280 workers created for large feature files is causing a 15 minute delay before the tests start.

What is your expected behavior?

If there are 3 scenarios tagged in 3 separate feature files, it should create 3 workers, not 30.

Please note, it is possible to create only one worker by adding an inner array to the array of spec files.

specs: [
        ['./src/features/**/*.feature'],
    ],

However I expect the behavior to be one worker per feature file so it is reported correctly at the end. See this article here: https://webdriver.io/blog/2019/11/01/spec-filtering/

How to reproduce the bug.

You can reproduce this in the cucumber boilerplate project here: https://github.com/webdriverio/cucumber-boilerplate

Steps to Reproduce:

1. Clone the above project
2. Run yarn install in the projects root directory
3. Change line 176 in the file cucumber-boilerplate/wdio.conf.ts from tagExpression: 'not @Pending', to tagExpression: '@test',.
4. Add the @tag tag to 3 different scenarios in 3 different feature files located in cucumber-boilerplate/src/features. The tag should be added in the line above the scenario name as such:

@test
    Scenario: Test if element responds to button press
        Given the element "#testKeyResponse" not contains any text
        When  I press "b"
        Then  I expect that element "#testKeyResponse" not contains the text "65"

5. In the root directory of the project run yarn test:features

Relevant log output

yarn run v1.22.10
$ wdio run wdio.BUILD.conf.ts

Execution of 30 workers started at 2022-04-19T21:06:27.792Z

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• [X] I have searched the existing issues

[christian-bromann]

Thanks for reporting!

Any contributions that resolves the bug are highly appreciated. Please take a look into our contribution guidelines and let us know if you have any questions. Cheers!

[julianfarizi]

Same Issue here. Using cucumber framework Framework will start more than 100 workers and 78 will be executed at the end issue about time will grow exponentially with the amount of feature files for sure. Any idea how to fix it? Happy to share more details if it will help to fix this behavior .. wondering that not more people report this issue :)

[christian-bromann]

Any idea how to fix it?

Please get involved and help us find a fix for this. Thanks!

[esnoar]

Any idea how to fix it?

Please get involved and help us find a fix for this. Thanks!

I'm facing this issue too and I'd be happy to contribute and help fix it. However, I don't have experience contributing to open-source projects and don't know what part of the code should I look at to debug this. Any suggestions?

[christian-bromann]

@esnoar as mentioned above the contribution guidelines is a good way to get started. Then I would take a look into the /packages/wdio-cucumber-framework/ directory that manages the Cucumber integration. In there is a method called _hasTests that probably is not working as expected. Feel free to ask question , also there are contributor office hours in case you want to pair with a core maintainer.

[esnoar]

@christian-bromann I debugged the code with the cucumber example located in examples/wdio/cucumber and I found that the method you mentioned is called after the workers have been initialized.

I dove deeper into other packages and found that the getSpecs() method defined in packages/wdio-config/src/lib/ConfigParser.ts is the one that is called before the workers initiate and it returns the spec files that will be executed.

This method is called in packages/wdio-cli/src/launcher.ts#L91 where the total number of workers is calculated.

My superficial analysis suggests fixing the issue by using this _hasTest() calculation in the getSpecs() method to verify that each of the spec files has at least one test that matches the tag filter. Then, a worker is initiated per each file that meets that condition. I think this can vary from framework to framework too.

As I mentioned before, I have very limited knowledge about JS/TS and OpenSource projects so I don't know if this would be a valid solution or even the best one. Let me know if I can do something else to contribute to this issue.

[christian-bromann]

Let me know if I can do something else to contribute to this issue.

Thanks for the analysis. It is hard to say what is going without debugging the issue. Unfortunately I don't have the capacity right now as I am working on the new v8 release. I am happy to support anyone who takes a stab at this.

[erwinheitzman]

@esnoar thank you for taking a look at this!

Since the issue seems quite specific to Cucumber, my guess would be that the tag filtering might not be working as expected which can be found here https://github.com/webdriverio/webdriverio/blob/c74ad391f1e8d353e9e7dbc78a3b1c857480604a/packages/wdio-cucumber-framework/src/cucumberEventListener.ts#L437

I think you are able to place a console.log and verify what the output of this is.

PS: I'm not saying you are wrong with your analysis, it's just a gut feeling that I have that it might be the above so I'm just sharing what I feel might be worth debugging :)

[esnoar]

@erwinheitzman thanks for the response.

I've created this branch to debug the issue.

I've edited the cucumber example by adding a new feature file that does not have any test that should execute depending on the tag @Skip.

https://github.com/esnoar/webdriverio/blob/debug/8253/examples/wdio/cucumber/features/my-feature-skip.feature

I've put a console.log in the following lines:

1. This one for knowing which spec files are used for starting the workers https://github.com/esnoar/webdriverio/blob/debug/8253/packages/wdio-config/src/lib/ConfigParser.ts#L243

2. This one to know the filtered pickles (the one you suggested) https://github.com/esnoar/webdriverio/blob/9b286a181d1e1bff2434ed27a91a2d9aa72b4d71/packages/wdio-cucumber-framework/src/cucumberEventListener.ts#L445

3. And this, for knowing what the hasTest value is for each feature https://github.com/esnoar/webdriverio/blob/debug/8253/packages/wdio-cucumber-framework/src/index.ts#L119

And other logs in the wdio-runner to follow the execution.

When I execute the example, I get the following log output:

[
  '/mnt/c/Users/enorena/Documents/projects/webdriverio/examples/wdio/cucumber/features/my-feature-skip.feature',
  '/mnt/c/Users/enorena/Documents/projects/webdriverio/examples/wdio/cucumber/features/my-feature.feature'
]

Execution of 2 workers started at 2022-07-13T13:49:55.601Z

2022-07-13T13:49:55.603Z INFO @wdio/cli:launcher: Run onPrepare hook
[
  '/mnt/c/Users/enorena/Documents/projects/webdriverio/examples/wdio/cucumber/features/my-feature-skip.feature',
  '/mnt/c/Users/enorena/Documents/projects/webdriverio/examples/wdio/cucumber/features/my-feature.feature'
]
2022-07-13T13:49:55.610Z INFO @wdio/cli:launcher: Run onWorkerStart hook
2022-07-13T13:49:55.611Z INFO @wdio/local-runner: Start worker 0-0 with arg: ./cucumber/wdio.conf.js
[0-0] 2022-07-13T13:50:12.205Z INFO @wdio/local-runner: Run worker command: run
[0-0] initialise worker
[0-0] initialise framework
[0-0] filtered pickles:
[0-0]
[0-0] has test: false
2022-07-13T13:50:38.953Z INFO @wdio/cli: [0-0] SKIPPED in chrome - /cucumber/features/my-feature-skip.feature
2022-07-13T13:50:38.953Z INFO @wdio/cli:launcher: Run onWorkerEnd hook
2022-07-13T13:50:38.954Z INFO @wdio/cli:launcher: Run onWorkerStart hook
2022-07-13T13:50:38.954Z INFO @wdio/local-runner: Start worker 0-1 with arg: ./cucumber/wdio.conf.js
[0-1] 2022-07-13T13:50:55.596Z INFO @wdio/local-runner: Run worker command: run
[0-1] initialise worker
[0-1] initialise framework
[0-1] filtered pickles: 
[0-1] 19,0,22,1,26,2
[0-1] has test: true
[0-1] RUNNING in chrome - /cucumber/features/my-feature.feature
[0-1] 2022-07-13T13:51:22.587Z INFO webdriver: Initiate new session using the WebDriver protocol
[0-1] 2022-07-13T13:51:24.965Z INFO webdriver: [POST] http://localhost:4444/session
[0-1] 2022-07-13T13:51:24.965Z INFO webdriver: DATA {
[0-1]   capabilities: { alwaysMatch: { browserName: 'chrome' }, firstMatch: [ {} ] },
[0-1]   desiredCapabilities: { browserName: 'chrome' }
[0-1] }
[0-1] filtered pickles: 
[0-1] 19,0,22,1,26,2
[0-1] 2022-07-13T13:51:29.223Z INFO webdriver: COMMAND navigateTo("https://github.com/")
[0-1] 2022-07-13T13:51:29.224Z INFO webdriver: [POST] 

// OMITTED THESE EXECUTION LOGS RELATED TO THE SCENARIOS

Spec Files:      0 passed, 1 failed, 1 skipped, 2 total (100% completed) in 00:01:44

2022-07-13T13:51:40.006Z INFO @wdio/local-runner: Shutting down spawned worker
2022-07-13T13:51:40.258Z INFO @wdio/local-runner: Waiting for 0 to shut down gracefully
2022-07-13T13:51:40.258Z INFO @wdio/local-runner: shutting down

I could notice that the filtering of the pickles is performed (properly) after the worker is started. Also, the first part of the logs indicates that both files were considered to start 2 workers as part of the getSpecs() method. That's why I think the tag filtering must be done before the workers start but I don't have a clue about how to implement that :/

Sorry for the poor formatted comment

Hope this helps!

[erwinheitzman]

@esnoar nice going with the debugging. A few things that I noticed:

1. I think that in order to skip the feature file you would have to put the tag at the feature level. I have tried that but it made no difference as of right now but when the feature is fixed, I do think this is needed.
2. I found the original implementation which might help to verify some assumptions https://github.com/webdriverio/webdriverio/pull/4531/files

By comparing the current code with the code from the original implementation I think we should be able to figure out what has been changed so that we can fix it :)

@christian-bromann it's a really big PR and I personally have no idea where to start looking at the moment, do you have any tips? I'd also like to know if I am making this bigger then it has to be by sharing this because I have a feeling I might be. It's a way of working that I often apply, when something is broken I go back and see what has changed ^^

So @esnoar no need to dive into the big PR I linked completely as I might be putting you on the wrong path but it might help at the same time as it's how this was originally implemented.

Again thanks for looking into this and don't worry about the formatting of your last message, it's looking fine to me and I have seen much much worse haha.

[christian-bromann]

It's a way of working that I often apply, when something is broken I go back and see what has changed ^^

I am doing the same thing. It is a good practice IMO.

do you have any tips?

As mentioned above, every framework adapter has a _hasTests method that checks if the worker would run any tests and if not, we would skip the session initiation and shutdown the worker immediately. So my assumption is that there is currently a bug within _hasTests maybe caused by a recent CucumberJS update.

[esnoar]

@erwinheitzman I think the feature should be skipped when not a single test matches the tag expression (which it skips properly but not at the ideal moment) defined in the config file and this decision should be computed before the worker starts.

I took a look at the PR and I noticed that this happens after the framework is initiated and the worker has started. This makes me think that the issue has been present since then.

So my assumption is that there is currently a bug within _hasTests maybe caused by a recent CucumberJS update.

To be honest, I think the _hasTest method is working properly but it is not being computed at the right moment to avoid the worker being initialized and wasting time by running the onPrepare and onWorkerStart hooks, initiating the framework and so on which takes approx 5 seconds per feature file that won't have any test executed (as shown in the logs I put before).

This is really impactful for projects that have a lot of feature files and even more if a tool like this is used for running scenarios in parallel 'cause it parses each scenario into a different feature file (which is my case).

[djv1001]

I'm currently experiencing this after upgrading from v7 to v8.

[sushantsoni5392]

This issue happens for jasmine framework as well when jasmine grep options are used.