Expected there to be a timestamp for current step error while running tests

[RankaPalak]

trafficstars

Current behavior

Random expected there to be a timestamp for current step (this might be a bug, please report at https://github.com/badeball/cypress-cucumber-preprocessor) error is occurring while running the tests. Because this error occurred during a after each hook we are skipping the remaining tests in the current suite:

Desired behavior

Test should not halt in between.

Test code to reproduce

-- To be added.

Versions

"@bahmutov/cypress-esbuild-preprocessor": "^2.2.0", "@deepakvishwakarma/cucumber-json-formatter": "^0.0.3", "cypress": "12.17.3", "cypress-recurse": "^1.31.2", "dayjs": "^1.11.7", "esbuild": "0.18.17", "pactum": "^3.2.3", "pkg": "^5.8.0", "prettier": "^2.8.0", "otplib": "^12.0.1", "cypress-wait-until": "^3.0.1" "@badeball/cypress-cucumber-preprocessor": "18.0.2", "convert-units": "^2.3.4", "cucumber-html-reporter": "^5.5.0", "cypress-keycloak-commands": "^1.2.0", "cypress-real-events": "^1.8.1", "dotenv": "16.3.1", "fs-extra": "^11.1.0", "lodash": "^4.17.15", "multiple-cucumber-html-reporter": "3.4.0", "node-fetch": "^2.6.0", "performance-results-parser": "0.0.4", "test-results-parser": "latest", "typescript": "^3.8.3"

• Node version: v18.14.2

Checklist

• [x] I've read the FAQ.
• [x] I've read instructions for logging issues.
• [x] I'm not using [email protected] (package name has changed and it is no longer the most recent version, see #689).

[badeball]

Hey, and thanks for reporting this issue. You need to provide a minimal, reproducible example that illustrates the issue.

[wakeboardfit]

We got the same last night. Unfortunately I don't have more information than already provided. This is the first time I've seen this though.

[RankaPalak]

This is random error and there is no way to reproduce this error. I am trying to create a repo with the required details in case I understand where exactly this issue is coming from.

[tbeckettaa]

I'm getting a similar but slight different error Expected to find a testStep...
Error: Unexpected state in afterSpecHandler: step-started ... that stops the test execution.

I've managed to reliably reproduce my issue by running 2 scenarios in a feature that both fail, with the second scenario changing the baseUrl and visiting a different domain. With Cypress.config("baseUrl", url); and cy.visit(url); I've found that if one of the tests passes, the error doesn't occur. Moving the scenarios into separate feature files also stops the error

@RankaPalak are you changing the baseUrl in your tests?

[badeball]

@tbeckettaa, aside from the somewhat similar wording of the error messages, each expectation-error usually represents completely different edge-cases, with uncomparable fixes. My advice is to open up a new ticket and provide said reproducible example.

[mscudlik]

Same issue here and some first analysis:

This is actually a symptom of the error handling here: https://github.com/badeball/cypress-cucumber-preprocessor/blob/e0ec8b2809dbfe73f40ac12b9cb249d7ef2c2cc3/lib/browser-runtime.ts#L949

This means that there is another error and because of that error no timestamp is there which then produces the error you see.

The other error is caused of a nullpointer accessing the id here:

https://github.com/badeball/cypress-cucumber-preprocessor/blob/e0ec8b2809dbfe73f40ac12b9cb249d7ef2c2cc3/lib/browser-runtime.ts#L546

This means the stack of is out of sync / empty.

[badeball]

@mscudlik, if you have seen line 545, IE.

https://github.com/badeball/cypress-cucumber-preprocessor/blob/e0ec8b2809dbfe73f40ac12b9cb249d7ef2c2cc3/lib/browser-runtime.ts#L545

.. fail with a "nullpointer access" (I assume you mean an error á Cannot read properties of undefined) then I'd love to see a reproducible example in order to remedy it.

[RankaPalak]

I'm getting a similar but slight different error Expected to find a testStep... Error: Unexpected state in afterSpecHandler: step-started ... that stops the test execution.

I've managed to reliably reproduce my issue by running 2 scenarios in a feature that both fail, with the second scenario changing the baseUrl and visiting a different domain. With Cypress.config("baseUrl", url); and cy.visit(url); I've found that if one of the tests passes, the error doesn't occur. Moving the scenarios into separate feature files also stops the error

@RankaPalak are you changing the baseUrl in your tests?

Hey @tbeckettaa No, I am not changing any URL.

[richardszanyi]

I'm having the same issue, it is randomly happening between 330 tests, so I don't think we would be able to provide a reproducible example..

Stacktrace on the error:

Because this error occurred during a `after each` hook we are skipping the remaining tests in the current suite: `Main Navigation` at createError (node_modules/@badeball/cypress-cucumber-preprocessor/dist/helpers/error.js:10:0) at fail (node_modules/@badeball/cypress-cucumber-preprocessor/dist/helpers/assertions.js:14:0) at assert (node_modules/@badeball/cypress-cucumber-preprocessor/dist/helpers/assertions.js:20:0) at assertAndReturn (node_modules/@badeball/cypress-cucumber-preprocessor/dist/helpers/assertions.js:23:0) at Context.afterEachHandler (node_modules/@badeball/cypress-cucumber-preprocessor/dist/browser-runtime.js:610:0) at Context.eval (node_modules/@badeball/cypress-cucumber-preprocessor/dist/browser-runtime.js:207:0)

[Takewkat]

Hey, yep the same issue, can not reproduce on local in CI the test doesn't even start, it crashes immediately on

• beforeEach in Test Replay
• afterEach in View Output

[badeball]

To everyone experiencing this error in a CI environment and not locally.

CI environments aren't actually inherently different and there's nothing that will only ever happen there. It's just a different machine. As explained multiple times on this tracker already, typically you do something different in CI.

You might be running $ cypress run in your CI environment, while $ cypress open locally. While they may seem similar, these are essentially two entirely different programs and comparing their results is almost nonsensical. Cypress does entirely different things when run headed vs. headless. This plugin will also behave different in these two scenarios.

You might be running Cypress in parallel in your CI environment, using something like the --parallel flag and you might think that this is something you can't replicate or do locally. However, you would be wrong. Using the --parallel flag is essentially the same as running a different program.

You might be running Cypress in a container in your CI environment. There's nothing stopping you from doing this locally as well.

You might be using a different configuration file when running Cypress in your CI environment, attaching things such as code coverage. This is also possible to do locally.

Essentially, there's no such thing a "this inherently only happens in a CI environment". You just haven't made a big enough effort to replicate the environment. And if you want to see this issue solved, you need to provide me with something that's going to help me reproduce it.

[grzegorzjarosz-securitas]

I appreciate your work maintaining this project @badeball , thanks !

Now, I learned that this error can happen due to multiple, different underlying issues. I'm adding this comment hoping it will help others with their investigations ;)

I've experienced it (due to bug in our code), while running Cypress tests with it's UI ( cypress open ), the result was that only first scenario passed successfully, any subsequent scenarios failed. When tests were executed in terminal (through cypress run), everything worked ( they all passed ).

In our case we were gathering executing command related informations (like timing, stack text to show in logs) by looking at command:start event. Note that following code leaves minimal example that reproduces the issue for our specific case:

Cypress.on("command:start", (c) => {
    // This one shows: 
    // "Expected there to be a timestamp for current step (this might be a bug, please report at https://github.com/badeball/cypress-cucumber-preprocessor)"
    if ( typeof c.attributes.args[0] === "function") {
        console.log(c.attributes.prev.userInvocationStack);
    }

    // this one shows right:
    // "Cannot read properties of undefined (reading 'userInvocationStack')" error
    // console.log(c.attributes.prev.userInvocationStack);
});

While code above includes our bug, it's still worth noting that exception thrown was hidden behind error returned by cucumber preprocessor making finding it harder.

Rest of the minimal example I prepared does not have anything speciall (Typescript (esbuild) project with single feature file with two scenarios, each with single step - with step implemented as empty code block). I can upload it to Github if needed.

[badeball]

@grzegorzjarosz-securitas, I was able to reproduce the issue. There's unfortunately no way (afaik) for the library to determine whether an error occurs in a step or "between" steps, as is happening when using Cypress.on. Hence, the library won't be able to maintain its statefulness required for reporting in this scenario.

@RankaPalak, are you doing something similar?

[badeball]

I've introduced a new flag in v23.2.0 to turn errors related to state tracking, as seen here, into soft errors - meaning that they won't halt execution, but rather an error is printed to stderr after the run.

This will hopefully ensure that errors aren't masked, as seen by @grzegorzjarosz-securitas's example. Thus possibly help users provide proper reproducible examples. Check the changelog for examples illustrating the configuration option.

-- To be added.

Since original poster seems absent and never posted any relevant code, I'm closing this. Feel free to open up another issue if anything changes in this regard.