Compatibility with Wazuh indexer based on OpenSearch 2.0.0

[gdiazlo]

Description

We need to ensure the UI compatibility with the next Wazuh indexer which might be based on OpenSearch 2.0.0. This update is still being discussed, and we need to know potential issues.

• [x] Review opensearch and opensearch-dashboard latest stable changelog
• [x] Identify improvements and potential impact on the UI
• [x] Report the findings to @wazuh/cicd to coordinate the upgrade effort.
• [x] Develop a testing environment to verify our components would work under this new build

Issues

• [x] #4480

[AlexRuiz7]

OpenSearch 2.0.0

Breaking changes

• Deprecate Compatibility override

The override main response setting compatibility.override_main_response_version is deprecated from OpenSearch version 1.x and removed from OpenSearch 2.0.0. This setting is no longer supported for compatibility with legacy clients.

• Deprecate outdated nomenclature

In order for OpenSearch to include more inclusive naming conventions, we’ve replaced the following terms in our code with more inclusive terms

• “Whitelist” is now “Allow list”
• “Blacklist” is now “Deny list”
• “Master” is now “Cluster Manager”

If you are still using the outdated terms in the context of the security APIs or for node management, your calls and automation will continue to work until the terms are removed later in 2022.

@wazuh/cicd These settings must be removed and updated in the opensearch.yml file.

Development environments

We need to adjust our development environments with the following changes:

• [x] Comment or remove the previous configuration setting in the opensearch.yml file
• [x] Re-tag our image on Quay.io from 2.0 to 2.0.0.
• [x] Fix a typo on our Docker Compose file, as the network is Docker network is created with the name os-dev-{OS_VERSION}. Must be changed to os-dev-${OS_VERSION} in order to name it dynamically.

Found issues

• [x] Our plugin contains custom styles for Kibana 7.9.0 which are causing troubles with the OSD 2.x styles. We need to review these styles and remove them if they are no longer required.

[AlexRuiz7]

OpenSearch 2.1.0 is live

OpenSearch 2.1.0 was released the 7th of July. We need to create images for this new version.

OpenSearch and Dashboards 2.1.0 Release Notes

https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.1.0.md

Tasks

• [x] Create and upload a Docker image for OSD-Dev 2.1.0 to our Quay.io repository.
• [x] Test our app with this new version.

[AlexRuiz7]

OpenSearch 2.2.0 is live

OpenSearch 2.2.0 was released the 12th of August. We need to create images for this new version, if we decide to provide support for this new version.

OpenSearch and Dashboards 2.2.0 Release Notes

https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.2.0.md

Tasks

• [x] Review the Release Notes for this version.

This version contains a CVE that is solved in 2.2.1. In case of providing support for OpenSearch 2.2.x we should upgrade directly to v2.2.1.

• [ ] Create and upload a Docker image for OSD-Dev 2.2.0 to our Quay.io repository.
• [ ] Test our app with this new version.

[AlexRuiz7]

Kind of? Depends on whether I create the quantity with pint or ureg:

...

That's not really ideal - force_ndarray_like is really a global choice about the behaviour of pint, it's not a choice specific to one unit registry...

I can definitely see that, but from what I recall every choice in Pint is on the registry level...it just happens to be that using the global Quantity class uses the application registry, which has a default, but otherwise can be configured with set_application_registry (xref https://github.com/xarray-contrib/pint-xarray/issues/7, https://github.com/hgrecco/pint/pull/880).

All that being said, I definitely agree about the repr. That would be a good fix!

What would be the fix though? If there was a QuantityArray class that would distinguish the reprs of the two objects, but barring that how else I am supposed to distinguish them? Unless you want to change the reprs to <Quantity(2.0, 'dimensionless')> and <Quantity(np.array(2.0), 'dimensionless')>?

I was guessing including the type of the magnitude in some form, a la Dask, so maybe <Quantity(2.0, units='dimensionless', type=float)> and <Quantity(2.0, units='dimensionless', type=numpy.ndarray)>? Though that might be misleading too since type isn't an argument to the constructor.

I personally don't really see what the point of diverging from numpy's interface at all is.

As a user who's only worked with Pint alongside NumPy, I would tend to agree, however, I recognize that the library has been deliberately designed for use without NumPy. But, with the current state of the ecosystem, maybe that's something that merits more discussion? What are the maintainers' thoughts here?

[yenienserrano]

In OpenSearch 2.2.1 a new CSS class is being added to the navbar components, in particular to the breadcrumbs headers, that breaks the visualization of the Wazuh breadcrumbs.

In the images below, we can see this new CSS class, named osdHeaderBreadcrumbs, which uses the filter property, root of these issues.

This problem applies to both, the light and the dark mode.

Disabling this property, considerably improves the situation, although there are rendering problems still.

[samrit-narshing]

hi . is there a way to know how to build or get the wazuh plugin package (zip) for opensearch 2.2.1 .. ? as i need to install plugin in existing opensearchv 2.2.1

[yenienserrano]

Hi @samrit-narshing thank you for your interest in Wazuh, currently the application is not compatible with OSD 2.2.1, this issue is to see if it can be compatible with this version.

[AlexRuiz7]

Hello @wazuh-cloud-cicd team.

One of the breaking changes of OS 2.x is the deprecation of the setting compatibility.override_main_response_version, however, we still use this setting in the 4.4 branch, with a comment that says that this setting is required for Filebeat 7.10.2 to work. This is worrying.

### Option to allow Filebeat-oss 7.10.2 to work ###
compatibility.override_main_response_version: true

Are you aware of this conflict?

[yenienserrano]

I was testing the configuration, and it only caused me problems in version 2.0.0, in versions 2.1.0 or 2.2.1 I had no problem.

The deprecation of the setting compatibility.override_main_response_version was reverted in version 2.1.0 and higher in this PR.

[AlexRuiz7]

OpenSearch 2.3.0 is live

OpenSearch 2.3.0 was released the 14th of September. We need to create images for this new version, if we decide to support it.

OpenSearch and Dashboards 2.3.0 Release Notes

https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.3.0.md

Tasks

• [x] Review the Release Notes for this version.
• [x] Create and upload a Docker image for OSD-Dev 2.3.0 to our Quay.io repository.
• [x] Test our app with this new version.

Notable changes

• Changes in some visualizations

:mountain_railway: Refactor

• [DeAngular][visualization][vislib] Remove angular from vislib (https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2138)
• Change timeline icon (https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2162)

[yenienserrano]

From what I was testing I could not find other things to highlight in the 2.2.1 versions and, since version 2.3.0 was released, I was testing that version and so far I found no differences in performance with respect to version 2.2.1.

[yenienserrano]

A new problem has been found in every version of OpenSearch 2.x, more concretely related to the OpenSearch Dashboards.

We use the flyout component from EUI, which closes automatically when the user clicks outside it. However, in 2.x, these components will also close automatically when clicking inside them. This is not the correct behavior, and it's probably related to OSD updating the EUI library to a newer version, breaking our customized components behavior.

Video

https://user-images.githubusercontent.com/63758389/191055783-8a658ce0-0bc4-466f-a050-f672f7a45360.mp4

We'll need to hunt and fix this bug.

Related release notes

OpenSearch Dashboards uses OUI and its alias onto EUI (https://github.com/opensearch-project/OpenSearch-Dashboards/pull/2080) https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/release-notes/opensearch-dashboards.release-notes-2.2.0.md

Bumps @elastic/eui to v34.6.0 and @elastic/charts to v31.1.0 https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/release-notes/opensearch-dashboards.release-notes-2.0.0.md

[yenienserrano]

Comparison table

Changes compared to OpenSearch and OpenSearch Dashboards 1.2.0.

	v2.0.0	v2.1.0	v2.2.1	v2.3.0
compatibility.override_main_response_version	not supported	supported	supported	supported
Changes in breadcrumb styles	No	No	Yes	Yes
Changes in the Elastic/EUI version	Yes, 34.6.0	Yes, 34.6.0	Yes, Fork 34.6.0	Yes, Fork 34.6.0
Flyouts issue	Yes	Yes	Yes	Yes
fixes CVE-2020-6383?	No	No	Yes	Yes
SAML login support*	Yes	No	No	Yes
SAML logoff support*	No	No	No	No

• SAML support is being investigated in this issue #4595

[AlexRuiz7]

The team has decided to move to OpenSearch 2.3.0 on the new minor version of Wazuh (4.4.0).