apm-agent-python icon indicating copy to clipboard operation
apm-agent-python copied to clipboard
verified publisher icon elastic

[CI] BDD Tests that are not implemented: should the build fail or warn?

Open mdelapenya opened this issue 4 years ago • 2 comments

We noticed that there are a few BDD files under https://github.com/elastic/apm-agent-python/tree/master/tests/bdd/features

  • api_key
  • azure_app_service_metadata
  • outcome

But only the API Key ones are implemented.

[2021-08-02T10:01:23.365Z] tests/bdd/init.py 0 0 0 0 100% [2021-08-02T10:01:23.365Z] tests/bdd/api_key_tests.py 31 15 0 0 52% [2021-08-02T10:01:23.365Z] tests/client/init.py 0 0 0 0 100%

You can check that they do not appear under the test list for that package: https://apm-ci.elastic.co/job/apm-agent-python/job/apm-agent-python-mbp/job/master/500/testReport/tests.bdd/

I'd like to ask about your expectations on it: should the build fail because they are not implemented, or show them as skipped?

mdelapenya avatar Aug 05 '21 08:08 mdelapenya

Show them as skipped, I think. Otherwise we'll start failing the moment that BDD tests are merged upstream, and I don't think that's the right behavior.

basepi avatar Aug 05 '21 14:08 basepi

Then I think the team needs to provide the scaffolding code for skipping the test using python's test runner. Otherwise they won't appear as skipped

mdelapenya avatar Aug 13 '21 07:08 mdelapenya