opensearch-api-specification icon indicating copy to clipboard operation
opensearch-api-specification copied to clipboard

[FEATURE] Add support for pending tests

Open dblock opened this issue 7 months ago • 0 comments

Is your feature request related to a problem?

In writing tests for _reindex I tried the following.

  - synopsis: Reindex from a remote cluster.
    path: /_reindex
    method: POST
    request_body:
      payload:
        source:
          remote:
            host: https://localhost:9200
            username: admin
            password: password
            socket_timeout: 30s
            connect_timeout: 30s
          index: movies
        dest:
          index: films
    response:
      status: 200
      payload:
        total: 1

This doesn't work because it needs a setting and an SSL cert.

I'd like to include the test, but mark it as pending, not run it, and show the explanation of why it wasn't run.

What solution would you like?

We had https://github.com/opensearch-project/opensearch-api-specification/pull/358 that would skip tests, but I'd actually like the test to run, but not fail.

In the test I want to write this:

  - synopsis: Reindex from a remote cluster.
    pending: |
       SSL needs to be configured to trust itself.
       For this to work, will need to:
       1. set reindex.remote.allowlist=localhost:9200 in docker-compose.yml
       2. setup SSL in a way that the docker instance trusts its own cert
    path: /_reindex
    method: POST

Then at runtime it should say

IGNORED (error, SSL needs to be configured to trust itself.)

If the test were to pass, it would actually fail because it shouldn't have been marked pending.

What alternatives have you considered?

  1. Comment out tests.
  2. https://github.com/opensearch-project/opensearch-api-specification/pull/358

dblock avatar Jul 11 '24 01:07 dblock