Add examples in test panel tree view

[alexvaut]

Is your feature request related to a problem? Please describe. The test panel is lacking the ability to run one example only.

Describe the solution you'd like Display in tree view the examples as well so it can be selected and run/debug.

Describe alternatives you've considered Run manually by editing launch.json

[jimasp]

are you talking about running a single example that sits inside a scenario outline?

please can you provide:

1. an example feature file, and
2. the behave command that you would run manually to achieve this? (the extension simply issues command line calls to behave.)

[alexvaut]

Here is a feature file

Feature: ESA Portal basic verification

  #noinspection CucumberUndefinedStep
  Scenario Outline: Verify <icon> icon is present on the ESA Portal.
    Given I am on the home page
    When I see the side menu bar is present
    Then I should find the icon "<icon>" with category "<category>"

    Examples:
      | icon              | category          |
      | Favorites         | favorites         |      
      | User Management   | user_management   |
      | System Management | system_management |

  Scenario Outline: Verify that <title> url is reachable.
    Given I am on the home page
    When I open the page at the required esa portal url /"<url>"
    Then I should find the title "<title>" with "<selector_type>" "<selector>"

    Examples: subscriber-search
      | url                                              | title              | selector_type | selector         |
      | trace/home/query?app=SUBSCRIBER_ACTIVITY_SEARCH  | Subscriber Search  | class         | top-page-title |
    Examples: network-search
      | url                                      | title          | selector_type | selector         |
      | trace/home/query?app=NETWORK_SEARCH      | Network Search | class         | top-page-title |
    Examples: call-search
      | url                                   | title                    | selector_type | selector         |
      | trace/home/query?app=CALL_SEARCH      | Call Search              | class         | top-page-title |

To verify that the "User Management" icon is present:

behave features/esa_portal_existence.feature -n "^Verify .* icon is present on the ESA Portal. -- @1.3"

1 is for the first list of examples and 3 is for the third element in this list which is the "User Management" icon.

To verify that the title network-search is fine

behave features/esa_portal_existence.feature  -n "^Verify that .* url is reachable. -- @2.1 .*"

2 is for the second list of examples and 1 is for the first element in this list which is the "Network Search" title. At the end, the regular expression is for the example list name that we don't care about (I believe).

[jimasp]

Ok, thank you, my initial thoughts are:

• This would be a non-trivial change (would take a while to implement and test) as every layer is affected here (parsing, ui tree, command line, output parsing, etc.). Also, most users would probably want this option off by default to cut down on UI noise, so there would probably be a need to add another extension setting (which also adds a bit more complexity + extra testing as you have different modes of operation).
• I'm not sure what would happen to the UI test panel tree if you have 100s of "Examples:" table lines in your project (which is not unusual). It's possible vscode performance could be impacted building the tree panel and it would certainly make the test panel tree noisy for large projects.
• I'm a little confused as to the use-case here, i.e. why it's useful to be able to do this in the IDE particularly vs just using a shell script to do the same thing. (Every feature that gets added makes the extension more complex, slower, and adds potential for bugs, so it has to be considered if there is a strong enough use-case for new features that could break more important things.)

[alexvaut]

most users would probably want this option off by default to cut down on UI noise

I worked quite a lot with Nunit and Visual Studio and this was supported by default:

just using a shell script to do the same thing

Everything can be replaced by command line calls but to be honest, having it in the tree makes it much easier to test and debug with just a click (vs. editing the arguments which is prone to errors, etc...)

[Marc2508]

You can have tags per example and therefore run single examples using tags if necessary.

[jimasp]

i actually wrote some code to implement this, but i haven't got around to testing it yet.