[v0.7] Elasticsearch Python client is incompatible with latest HA Version

[mike12806]

Environment Home-Assistant version: 2024.12.0 Elasticsearch version: 8.16.1

Relevant configuration.yml settings:

# Do not include your Elasticsearch URL, credentials, or any other sensitive information
elastic:
  ...

Describe the bug

        ^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1085, in _get_component
    raise ImportError(f"Exception importing {self.pkg_path}") from err
ImportError: Exception importing custom_components.elasticsearch

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1074, in _get_component
    ComponentProtocol, importlib.import_module(self.pkg_path)
                       ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/loop.py", line 200, in protected_loop_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.13/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1022, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/config/custom_components/elasticsearch/__init__.py", line 43, in <module>
    from .es_integration import ElasticIntegration
  File "/config/custom_components/elasticsearch/es_integration.py", line 11, in <module>
    from .es_doc_publisher import DocumentPublisher
  File "/config/custom_components/elasticsearch/es_doc_publisher.py", line 15, in <module>
    from custom_components.elasticsearch.es_index_manager import IndexManager
  File "/config/custom_components/elasticsearch/es_index_manager.py", line 6, in <module>
    from elasticsearch7 import ElasticsearchException
  File "/usr/local/lib/python3.13/site-packages/elasticsearch7/__init__.py", line 36, in <module>
    from .client import Elasticsearch
  File "/usr/local/lib/python3.13/site-packages/elasticsearch7/client/__init__.py", line 22, in <module>
    from ..transport import Transport, TransportError
  File "/usr/local/lib/python3.13/site-packages/elasticsearch7/transport.py", line 25, in <module>
    from .serializer import JSONSerializer, Deserializer, DEFAULT_SERIALIZERS
  File "/usr/local/lib/python3.13/site-packages/elasticsearch7/serializer.py", line 50, in <module>
    np.float_,
    ^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/numpy/__init__.py", line 400, in __getattr__
    raise AttributeError(
    ...<3 lines>...
    )
AttributeError: `np.float_` was removed in the NumPy 2.0 release. Use `np.float64` instead.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 334, in _async_setup_component
    component = await integration.async_get_component()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1034, in async_get_component
    self._component_future.result()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1026, in async_get_component
    comp = self._get_component()
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1085, in _get_component
    raise ImportError(f"Exception importing {self.pkg_path}") from err
ImportError: Exception importing custom_components.elasticsearch[0m

Expected behavior A clear and concise description of what you expected to happen.

Expect Elasticsearch integration to start

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.

[legrego]

Thanks for raising this, @mike12806. It looks like there is an incompatibility with the bersion of the ES Client we're using in the 0.7.x line of this component.

@sorenlouv discovered that updating to version 2 of our component fixes the issue, but there are significant changes between 0.7 and 2.0.

https://github.com/legrego/homeassistant-elasticsearch/releases/tag/v2.0.0-RC2

[dwissm1]

I jumped up to 2.0 after .7 went haywire and no matter what I do I cannot disable Verify TLS certificate (recommended) and Verify hostname matches server certificate. It forces me to enable them but I'm running on a local network and don't care about it. Just want the logs.

Any ideas?

Usually in the code I do:

verify_certs=False, ssl_show_warn=False,

I extracted the CRT for the CA for my Elastic instance, still no no joy. (gave it the path: /config/certs/ca.crt) I added a fully qualified path, made sure it was 644. Tried to switch off VERIFY TLS AND Verify Hostname and all combinations of both. Still nothing.

Elastic version 8.16.1 HA: Core 2024.12.1 Supervisor 2024.11.4 Operating System 14.0 Frontend 20241127.6

Log Details: 2024-12-09 09:57:16.595 INFO (MainThread) [custom_components.elasticsearch] Entering custom_components.elasticsearch.config_flow : ElasticFlowHandler.async_step_certificate_issues 2024-12-09 09:57:16.597 INFO (MainThread) [custom_components.elasticsearch] Returning from custom_components.elasticsearch.config_flow : ElasticFlowHandler.async_step_certificate_issues 2024-12-09 09:57:16.661 INFO (MainThread) [custom_components.elasticsearch] Entering custom_components.elasticsearch.config_flow : ElasticFlowHandler.async_step_certificate_issues 2024-12-09 09:57:16.661 INFO (MainThread) [custom_components.elasticsearch] Returning from custom_components.elasticsearch.config_flow : ElasticFlowHandler.async_step_certificate_issues 2024-12-09 09:57:18.024 ERROR (MainThread) [custom_components.nest_protect] Unknown exception. Please create an issue on GitHub with your logfile. Updates paused for 5 minutes. Traceback (most recent call last): File "/config/custom_components/nest_protect/init.py", line 209, in _async_subscribe_for_data dict(b, **buckets.get(b.object_key, {})) for b in [data.updated_buckets] ^^^^^^^^^^^^ AttributeError: 'list' object has no attribute 'object_key'

[mike12806]

Thanks for raising this, @mike12806. It looks like there is an incompatibility with the bersion of the ES Client we're using in the 0.7.x line of this component.

@sorenlouv discovered that updating to version 2 of our component fixes the issue, but there are significant changes between 0.7 and 2.0.

https://github.com/legrego/homeassistant-elasticsearch/releases/tag/v2.0.0-RC2

What type of significant changes? I'm using a pretty bog standard Elasticsearch setup and trying to connect without auth over http....

This is with v2.0.0-RC3

2024-12-06 00:05:40.091 INFO (MainThread) [custom_components.elasticsearch] Entering custom_components.elasticsearch.config_flow : ElasticFlowHandler.async_step_user
2024-12-06 00:05:40.091 INFO (MainThread) [custom_components.elasticsearch] Returning from custom_components.elasticsearch.config_flow : ElasticFlowHandler.async_step_user
2024-12-06 00:05:59.164 INFO (MainThread) [custom_components.elasticsearch] Entering custom_components.elasticsearch.config_flow : ElasticFlowHandler.async_step_user
2024-12-06 00:05:59.165 INFO (MainThread) [custom_components.elasticsearch] Returning from custom_components.elasticsearch.config_flow : ElasticFlowHandler.async_step_user
2024-12-06 00:05:59.177 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/config/custom_components/elasticsearch/es_gateway_8.py", line 302, in _error_converter
    yield
  File "/config/custom_components/elasticsearch/es_gateway_8.py", line 196, in has_privileges
    response = await self.client.security.has_privileges(**privileges)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/elasticsearch8/_async/client/security.py", line 1811, in has_privileges
    return await self.perform_request(  # type: ignore[return-value]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
    )
    ^

[strawgate]

The 2.0 integration switches to using data streams by default and changes the schema and mapping to take full advantage of data types in Elasticsearch (among other things).

You will have to update the permissions for your home assistant use following the setup instructions in the wiki.

You'll also have to update visualizations to use the new mappings.

[strawgate]

I jumped up to 2.0 after .7 went haywire and no matter what I do I cannot disable Verify TLS certificate (recommended) and Verify hostname matches server certificate. It forces me to enable them but I'm running on a local network and don't care about it. Just want the logs.

Any ideas?

Usually in the code I do:

verify_certs=False, ssl_show_warn=False,

Can you file a separate issue with this? Can you include debug logs of you attempting to set this up?

[mike12806]

https://github.com/legrego/homeassistant-elasticsearch/issues/399 - with debug logs

[strawgate]

Looks like fixing this on the 0.7.x branch would require bumping the ES7 dependency to 7.17.12 https://github.com/elastic/elasticsearch-py/releases/tag/v7.17.12

That may break compatibility with 7.11 to 7.16.

As an alternative we could bump the 1.x branch to minimum 7.17 and then release the 1.x branch

[bdauvissat]

Since my last update of HA, I have quite a similar error, the logs are a little bit different. My current version :

I also tried to remove the integration (successfully) and add it again and then, I have an error message

[strawgate]

Since my last update of HA, I have quite a similar error, the logs are a little bit different. My current version :

I also tried to remove the integration (successfully) and add it again and then, I have an error message

Hi,

Can you file a new issue with your information? Can you include the version of the integration as well as the full text from the exception on line one?

[bdauvissat]

Sure. I thought it was the same issue and didn't want to overflow you with similar issues. Sorry

[strawgate]

Hi, we have published a new pre-release of the v1.0.0 branch that should resolve this issue. Can you switch to that version? For others watching this issue, please note the minimum Elasticsearch version has changed to 7.17 for the v1.0.0 branch.

[mike12806]

Hi, we have published a new pre-release of the v1.0.0 branch that should resolve this issue. Can you switch to that version? For others watching this issue, please note the minimum Elasticsearch version has changed to 7.17 for the v1.0.0 branch.

Sure - I have things working now on v2.0.0-RC4 but I can try to swap back and see if things work on the new v1 branch in a bit

[strawgate]

Staying on the 2.0.0 branch is fine too :) Swapping back while pointed to the same destination ES cluster may complicate things