Comment object no longer included in the JSON output

[estucklesskrikey]

Describe the bug Comment object has been removed from the -f JSON output or when we convert it using the newly recommended json convertor. Before this was removed, a given step would include any and all lines above it that were comments.

To Reproduce Steps to reproduce the behavior:

    Given I select the settings button
    # testcase-11111
    And I enter email 
    # testcase-22222 testcase-33333
    Then I should see the summary text under username and password fields
    Then I enter new account information
    And I tap the navigation tab video button
    # testcase-44444 testcase-55555
    Then I should see the video feed 

1. Run a scenario that looks like above
2. Open json or message results data
3. Notice how the comments aren't encapsulated in the results data

Expected behavior I would expect something like the following to exist in the results data

{"keyword"=>"Then ",
        "name"=>"I should see the video feed",
        "line"=>16,
         "comments" => [ "# testcase-44444 testcase-55555" ],
        "match"=>
         {"location"=>"(removed)"},
        "result"=>{"status"=>"passed", "duration"=>3262485000}}

Context & Motivation

The comments allowed us to encapsulate the testrail ID of a given test step. Then we could identify which testrail cases passed in a scenario even if a later step failed. Now that this is removed, our tracking in this way no longer functions.

Screenshots If applicable, add screenshots to help explain your problem.

Your Environment

• Versions used [e.g 5.3.0]

[aurelien-reeves]

It looks somehow related to https://github.com/cucumber/cucumber/issues/1364

I think new versions of cucumber does not process comment at all by design. If you want to add documentation to you features and scenarios, description should be helpful.

What would be the purpose of reporting comments in the feature file?

[mattwynne]

What would be the purpose of reporting comments in the feature file?

@aurelien-reeves I think that @estucklesskrikey explained his motivation here:

The comments allowed us to encapsulate the testrail ID of a given test step

As to the root cause of this, I see a few test-cases for the new Gherkin parser that mention comments, but we'd need to study it a bit more to check whether any of them cover the case of a commented-out step. It could be that we just forgot to test/build for this case.

[estucklesskrikey]

For more clarification, I'll refer to my above example scenario:

    # testcase-11111
    And I enter email 
    # testcase-22222 testcase-33333
    Then I should see the summary text under username and password fields
    Then I enter new account information
    And I tap the navigation tab video button
    # testcase-44444 testcase-55555
    Then I should see the video feed  

In the case that this scenario runs, but fails during the step: "Then I enter new account information", the scenario would resolve logically like the following:

    # testcase-11111
    And I enter email (PASSED)
    # testcase-22222 testcase-33333
    Then I should see the summary text under username and password fields (PASSED)
    Then I enter new account information (FAILED) 
    And I tap the navigation tab video button (SKIPPED)
    # testcase-44444 testcase-55555
    Then I should see the video feed  (SKIPPED)

In our automation repo, we integrate with a test case repository to generate reports of each run. In the above case, we would mark testcase-11111, testcase-22222, and testcase-33333 as PASSED. While we'd mark testcase-44444, and testcase-55555 as SKIPPED. This extra bit of info provides a lot of help across our many test cases; reducing debugging time, and giving us a better representation of the true failure rate of the app under test.

[aurelien-reeves]

Comments are still part of the messages, they still exist, and are part of the message formatter.

That mean that if you use --format message --out report.ndjson, you will have a report with step comments in it. It is not as human friendly as the json formatter, but maybe that may help you with your issue with some scripting?

Also, that mean that maybe it is possible to update the json converter to bring back those comments.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two months if no further activity occurs.

[luke-hill]

Given the JSON formatter is in maintenance mode and actively discouraged from usage and we have the object/s in messages I'm going to close this as will not fix.

cc/ @estucklesskrikey