wagtail
OpenSearch as replacement for propietary ElasticSearch
Is your proposal related to a problem?
Yes. I have requirements to use Open Source infrastructure, as we cannot risk vendor lock-in for core components.
Describe the solution you'd like
Full support for OpenSearch, https://www.opensearch.org/ a fork of the open source version of ElasticSearch after then went proprietary on their license.
Describe alternatives you've considered
Use an old ElasticSearch Create my own OpenSearch integration
Additional context
ElasticCo changed the license of new versions of their products to a proprietary, source available license https://www.elastic.co/pricing/faq/licensing
The Wagtail marketing website refers to ElasticSearch in several locations as being Open Source, however, this is no longer the case and the marketing website should be updated to reflect the Source Available license change in ElasticSearch. Preferably, an new release of Wagtail would switch to using the Open Source OpenSearch and the marketing would be updated to talk about OpenSearch instead of ElasticSearch.
As a community, we can be thankful for the Open Source work that Elastic has created in the past, and their continued contributions to Open Source projects like Apache Solar, however, we should not continue to refer to ElasticSearch as Open Source when that is no longer the case. They as a company made a choice to switch to a proprietary non-open license, and with that choice comes consequences. We as a community need to react accordingly to not spread confusion about what is and is not open source. The simplest and best approach is to adopt OpenSearch as the standard search interface in Open Source projects like wagtail. In the meantime, references to Elastic being open source should be removed from the website and marketing materials.
Hi @ensignavenger, thank you for the suggestion. This sounds to me like a welcome addition. In our docs, I see we explicitly mention support for Amazon AWS Elasticsearch Service, which I believe is now Amazon OpenSearch Service, so it could be the existing search backend would already support OpenSearch with little to no change? This seems to be using our own wagtail.search.backends.elasticsearch5 backend.
If a new backend has to be built, I have a feeling it might be easier to build this as an external package first (see for example wagtail-whoosh), since Wagtail has good support for creating and using external search backends.
For the marketing website, I saw two mentions on the Features page as you described:
Elasticsearch is an excellent open source search engine with straightforward integration with Wagtail and great performance.
And:
Elasticsearch Wagtail uses Elastic for finding content within the admin interface. It provides quick and fluent access to pages, images and documents. We use Elastic because it’s open source, gives great performance and offers a straightforward integration with Wagtail.
I’m not too sure what I’d replace those with myself - someone else might feel more inspired, if not I’ll let people who look after the site know they’ll have to change this.
Thank you for responding and for your work providing Wagtail! I agree it probably would not be too much work to get Open Search working with minimal changes to the existing integration code. I do have some concern for the future maintenance as development diverges between Elastic and OpenSearch. I haven't had a chance to dig in to see what it would take myself, yet.
As for the marketing site, I think those are the two references I found as well- simply removing the reference to elastic being Open Source would be a good first pass. It could be removed entirely or changed to "source available".
We use Elastic because it provides great performance and offers a straightforward integration with Wagtail. Wagtail also supports alternative search back ends via packages...
FYI, I tried dropping in OpenSearch in place of Elasticsearch, but get:
$ ./manage.py wagtail_update_index
Updating backend: default
default: Rebuilding index wagtail__wagtailcore_page
Traceback (most recent call last):
…
File "/…/venv/lib/python3.10/site-packages/elasticsearch/transport.py", line 421, in perform_request
_ProductChecker.raise_error(self._verified_elasticsearch)
File "/…/venv/lib/python3.10/site-packages/elasticsearch/transport.py", line 638, in raise_error
raise UnsupportedProductError(message)
elasticsearch.exceptions.UnsupportedProductError: The client noticed that the server is not Elasticsearch and we do not support this unknown product
$ echo \U+1F615
😕
FYI, more FOSS organizations are dumping Elasticsearch. For example, Homebrew won't install it, reports:
$ brew info elasticsearch
==> elasticsearch: stable 7.10.2
Distributed search & analytics engine
https://www.elastic.co/products/elasticsearch
Disabled because it is switching to an incompatible license. Check out `opensearch` instead!
Not installed
…
Previously, I've had luck supporting OpenSearch using the elasticsearch7 backend and pinning the elasticsearch dependency to 7.13.4 (the last version before the es library started throwing the UnsupportedProductError when connecting to a non-Elastic product). However, in Wagtail 5.1 (commit 9203716bae8c55b460e309b16c65d5311ad6c747 ) the ES7 backend has started including settings as a keyword argument in its calls for compatibility for elasticsearch versions >= 7.15 or so. Running on an older version of elasticsearch now gives an exception like:
Traceback (most recent call last):
File "<project root>/manage.py", line 23, in <module>
execute_from_command_line(sys.argv)
File "<project root>/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
utility.execute()
File "<project root>/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "<project root>/venv/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
self.execute(*args, **cmd_options)
File "<project root>/venv/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
output = self.handle(*args, **options)
File "<project root>/venv/lib/python3.9/site-packages/wagtail/search/management/commands/update_index.py", line 158, in handle
self.update_backend(
File "<project root>/venv/lib/python3.9/site-packages/wagtail/search/management/commands/update_index.py", line 84, in update_backend
index = rebuilder.start()
File "<project root>/venv/lib/python3.9/site-packages/wagtail/search/backends/elasticsearch5.py", line 984, in start
self.reset_index()
File "<project root>/venv/lib/python3.9/site-packages/wagtail/search/backends/elasticsearch5.py", line 980, in reset_index
self.index.reset()
File "<project root>/venv/lib/python3.9/site-packages/wagtail/search/backends/elasticsearch5.py", line 972, in reset
self.put()
File "<project root>/venv/lib/python3.9/site-packages/wagtail/search/backends/elasticsearch7.py", line 25, in put
self.es.indices.create(index=self.name, **self.backend.settings)
File "<project root>/venv/lib/python3.9/site-packages/elasticsearch/client/utils.py", line 168, in _wrapped
return func(*args, params=params, headers=headers, **kwargs)
TypeError: create() got an unexpected keyword argument 'settings'
As far as I can tell, there's no way to run Wagtail 5.1 on OpenSearch (and in particular Amazon OpenSearch Service) at this point.
Marking this for a potential 5.1.2 bugfix release, since if there was a previously working configuration (even if it's not one that we knew about :-) ) then this can be considered a regression.
We're seeing the same error IndicesClient.create() got an unexpected keyword argument 'settings' when using a Bonsai Elasticsearch 7.10.2 instance.
Current Django/Wagtail versions are:
Django==4.2.4
wagtail==5.1.1
Previous versions are working fine:
Django==4.1.7
wagtail==4.2.1
In both environments the local elasticsearch is elasticsearch==7.13.4
Have fixed compatibility with <7.15 versions of the library in #10916, and added Opensearch to our CI. Just needs adding documentation, and I think we can call this officially supported.