Sniffing ignores authorization options and x-opaque-id

[KACAH]

Elasticsearch version (bin/elasticsearch --version): Version: 8.2.2, Build: default/docker/9876968ef3c745186b94fdabd4483e01499224ef/2022-05-25T15:47:06.259735307Z, JVM: 18.0.1.1

elasticsearch-py version (elasticsearch.__versionstr__): 8.2.3

Sniffing callback uses Transport instance directly to perform /_nodes/_all/http requests. https://github.com/elastic/elasticsearch-py/blob/0da2ba901514447089f8a290f0ddba4f18cf53f8/elasticsearch/_async/client/_base.py#L172

However, both Authorization and opaque-id headers are NOT passed to Transport class during initialization. https://github.com/elastic/elasticsearch-py/blob/0da2ba901514447089f8a290f0ddba4f18cf53f8/elasticsearch/_async/client/init.py#L324

Instead they are saved to Elasticsearch client instance after Transport is already created https://github.com/elastic/elasticsearch-py/blob/0da2ba901514447089f8a290f0ddba4f18cf53f8/elasticsearch/_async/client/init.py#L407

and used for API requests only. https://github.com/elastic/elasticsearch-py/blob/0da2ba901514447089f8a290f0ddba4f18cf53f8/elasticsearch/_async/client/init.py#L407

As the result sniffing fails with Exception elastic_transport.SniffingError: No viable nodes were discovered on the initial sniff attempt while the real problem is

{
   "error":{
      "root_cause":[
         {
            "type":"security_exception",
            "reason":"missing authentication credentials for REST request [/_nodes/_all/http]",
            "header":{
               "WWW-Authenticate":[
                  "Basic realm=\"security\" charset=\"UTF-8\"",
                  "ApiKey"
               ]
            }
         }
      ],
      "type":"security_exception",
      "reason":"missing authentication credentials for REST request [/_nodes/_all/http]",
      "header":{
         "WWW-Authenticate":[
            "Basic realm=\"security\" charset=\"UTF-8\"",
            "ApiKey"
         ]
      }
   },
   "status":401
}

The problem is the same for both Sync and Async clients.

The fix might be to build opaque-id and authorization headers before transport intialization to and pass them to client_node_configs method (headers argument). However, I am not sure if it will work "as intended" with .options() feature and internal _transport constructor argument of Elasticsearch and AsyncElasticsearch classes.

[VanillaDevelop]

This appears to still be an issue on elasticsearch-py version 8.5.0 and elasticsearch version 8.5.3. The misleading error message cost us quite some time re-evaluating our elasticsearch configuration. Would be nice if someone could take a look at it.

[KristobalJunta]

I have experienced the same issue. And indeed it

cost us quite some time re-evaluating our elasticsearch configuration

due to the observed behavior being:

• client starts with a proper configuration
• after some time (appeared to be related to sniff timeout) it suddenly starts throwing 401 errors

Client was configured using RFC-1738 formatted URL, which is one of the officially proposed and supported options.

connections.configure(
    default={
        "hosts": [settings.ES_URL],  # URL formatted as "http://user:password@host:port"
        "retry_on_timeout": True,
        "sniffer_timeout": 3600,
    }
)

Nevertheless, after sniffing was done, connection pool lost all auth info and attempted to make unauthenticated requests.

The solution was either a. disable sniffing b. pass auth params explicitly:

from yarl import URL

es_url = URL(settings.ES_URL)  # here ES_URL is parsed into components

connections.configure(
    default={
        "hosts": [str(es_url.origin())],  # "host:port"
        "http_auth": f"{es_url.user}:{es_url.password}",
        "retry_on_timeout": True,
        "sniffer_timeout": 3600,
    }
)

I find this behavior misleading and not intuitive. It is also not described anywhere seemingly.

[redbaron4]

Just ran into this today. Our Elasticsearch API is exposed via an Nginx reverse proxy protected by basic auth. The reverse proxy itself uses HTTPS with self signed certificate

Trying to instantiate Elasticsearch with

Elasticsearch("https://internal.foo.com:443/elknew", sniff_on_connection_fail=True, sniff_on_start=True, max_retries=10, retry_on_timeout=True, basic_auth=["username", "password"], verify_certs=False)

fails with

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lib/python3.6/site-packages/elasticsearch/_sync/client/__init__.py", line 398, in __init__
    **transport_kwargs,
  File "lib/python3.6/site-packages/elastic_transport/_transport.py", line 246, in __init__
    self.sniff(True)
  File "lib/python3.6/site-packages/elastic_transport/_transport.py", line 450, in sniff
    "No viable nodes were discovered on the initial sniff attempt"
elastic_transport.SniffingError: No viable nodes were discovered on the initial sniff attempt

And nginx logs show that 401 was raised.

However, if we instantiate Elasticsearch with sniff_on_start=False, it works correctly.

[siyanew]

Is there any way to have sniff_on_start=True and send authorization to sniff request? Any update or fix?

[jhopkins219]

Howdy, I ran into this problem upgrading our clients from 7.x to 8.8 and wrote a workaround by overwriting the internal callback that does these sniff requests. Hope this issue can be addressed soon, but in the meantime you can still use the sniff options by initializing the connection like so (note this is for the sync client, and may need changes depending on your client version and needs):

https://gist.github.com/jhopkins219/2b5b38061f1b87515c7a5c29ca91a0a3

[xkobal]

@jhopkins219 Thanks for your fix. I have tried it and it works with some adaptation (I need api key in my case).

But next step is a new problem, direct queries on nodes seems to lost my ca_certs configuration and throw this error:

elastic_transport.TlsError: TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'https://NODE_HOSTNAME:9200'. (_ssl.c:1007))

I'm currently investigating in URLlilb, HttpRequest from elastic_transport but without success.

[acidvegas]

Bumping cause this 2022 issue is still persistent.....classic enterprise devs lol

[acidvegas]

Howdy, I ran into this problem upgrading our clients from 7.x to 8.8 and wrote a workaround by overwriting the internal callback that does these sniff requests. Hope this issue can be addressed soon, but in the meantime you can still use the sniff options by initializing the connection like so (note this is for the sync client, and may need changes depending on your client version and needs):

https://gist.github.com/jhopkins219/2b5b38061f1b87515c7a5c29ca91a0a3

I can confirm @jhopkins219 patch fixes this issue for me! Props

[prsinghs]

is this fix released? facing the same on 8.12 elasticsearch-py

[acidvegas]

is this fix released? facing the same on 8.12 elasticsearch-py

nope. lazy enterprise devs still havent paid attention to this 2 year old issue.....

https://github.com/elastic/elasticsearch-py/blob/8.12/elasticsearch/_sync/client/_base.py#L166 ^ missing the auth header entirely still. Use the patch @jhopkins219 shared, it works perfectly.

I have also sent in a PR for a potential fix to this issue...but with this issue being from 2022, who knows when the developers will push this to the upstream.