grepTags and grepOmitFiltered:true runs skipped tests

[sivapriya-k]

When grepTags and grepOmitFiltered:true are used together, skipped tests ( via it.skip) are not skipped.

When a test is skipped via it.skip, it is not skipped when grepOmitFiltered:true is used in conjunction with grep tags.

[bahmutov]

Can you provide an example with exact tests titles and tags and how you run them? Also read the section on debugging in the readme

Sent from my iPhone

On Nov 16, 2021, at 19:21, sivapriya-k @.***> wrote:

 When grepTags and grepOmitFiltered:true are used together, skipped tests ( via it.skip) are not skipped.

When a test is skipped via it.skip, it is not skipped when grepOmitFiltered:true is used in conjunction with grep tags.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[sivapriya-k]

The test structure is describe('Page> List', () => { describe('List', { tags: ['us1'] }, () => { it.skip("test name", ....

this is the env settings

And I'm running via cypress open. When I run this spec, the skipped test is run

[bahmutov]

And what tests does it run?

Sent from my iPhone

On Nov 16, 2021, at 19:31, sivapriya-k @.***> wrote:

 The test structure is describe('Page> List', () => { describe('List', { tags: ['us1'] }, () => { it.skip("test name", ....

this is the env settings

And I'm running via cypress open

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[sivapriya-k]

It runs only the skipped test, other tests in the describe block (which are not skipped) are not run

[bahmutov]

I have added a test in https://github.com/cypress-io/cypress-grep/commit/13bdac701ee8992b9fbb711e057ffcbc79ad1ac4 and do not see anything unusual. What do you think is wrong?

[yrollam]

when we have a flaky test, we utilize it.skip to ensure the tests are not being run until someone has time to address the failure. the problem we’re seeing is that any files with an it.skip are being picked up in team workflows that the skipped tests are not tagged with. in the example below, the test is tagged with @voyager and it is not run (as expected) with the voyager job. however, the test is picked up in the @spork job where the test setup is run (the setup is read from the context, where persona defines which beforeEach functions run) and the test is ultimately marked as pending.

this is an issue for us because our setup (again, defined by the persona in the context) is complex/extensive setup to create all the test data we need for each test. having this setup run in a group where it shouldn’t, for the test to ultimately be skipped, causes a lot of issues for our team. if a test describe block is tagged with a team and an it block is skipped inside that file, how can we avoid it being picked up with other tagged files?

we are utilizing tags to identify which dev team owns a test. we then use the tags to group tests in our circle ci config. below i’ve included an example of the anchors we’re using, the workflow for running our tests, as well at the test file mentioned above.

anchor:

cypress-next-spork: &cypress-next-spork
  name: Spork/VoC - Regression Next
  env: 'grepTags=@spork,grepOmitFiltered=true'
cypress-next-voyager: &cypress-next-voyager
  name: Voyager - Regression Next
  env: 'grepTags=@voyager,grepOmitFiltered=true'

workflow in the yaml file:

  cypress-next-regression:
    when:
      and: [<<pipeline.parameters.run_cypress_next_reg>>, <<pipeline.parameters.api_trigger>>]
    jobs:
      - cypress/install:
          <<: *cypress-install-dev
      - cypress/run:
          <<: *cypress-run-dev-remote
          <<: *cypress-next-voyager
          parallelism: 2
          spec: cypress/integration/next/regression/**/*
          config-file: cypress/environment-configs/dev.json
          group: 'Voyager Regression [Next] - Dev'
          tags: 'voyager,regression-next,dev'
          post-steps:
              - run:
                  <<: *cypress-next-cleanup
      - cypress/run:
          <<: *cypress-run-dev-remote
          <<: *cypress-next-spork
          parallelism: 2
          spec: cypress/integration/next/regression/**/*
          config-file: cypress/environment-configs/dev.json
          group: 'Spork Regression [Next] - Dev'
          tags: 'spork,regression-next,dev'
          post-steps:
            - run:
                <<: *cypress-next-cleanup

test with it.skip:

import Dashboard from '../../../../../pages/dashboards/dashboard.page';
import { getEnvUrlWithSubId } from '../../../../../support/utils';

describe('Load Dashboard page', { tags: '@voyager' }, () => {
    context(
        'as a free user',
        {
            env: {
                persona: 'free'
            }
        },
        () => {
            it.skip('C12995 should load /dashboards', () => {
                cy.visit('/');
                cy.mockAgent();
                cy.url().should('equal', `${getEnvUrlWithSubId()}/`);
                Dashboard.titleContainer().click();
                Dashboard.pageTitle().should('contain', 'Dashboard');
            });
        }
    );
});

Our temporary solution was going to be tagging the tests with @skip, but that means tagging all it blocks in a file rather than the main describe block, so it's not ideal. We ended up commented out the tests.

[jordanpowell88]

We are in the process of deprecating this repo as this package has been republished under @cypress/grep and has moved to https://github.com/cypress-io/cypress/tree/develop/npm/grep. I'm going to go ahead and close this issue as it appears stale. If this is still an issue, please open a new issue here