See if we can update to Elasticsearch 8.0

[humphd]

Elasticsearch 8.0 was recently released: https://www.elastic.co/blog/whats-new-elastic-8-0-0. It just made it to Docker Hub too: https://hub.docker.com/_/elasticsearch.

@manekenpix tells me that every time we update Elasticsearch it breaks, so we can't do this without testing. Luckily, @rclee91 and @JiaHua-Zou are landing tests for Elasticsearch in 2.7, so soon we can try this.

[cindyorangis]

It's either Elasticsearch or MDX that breaks Telescope, eh? 😂

[DukeManh]

Once @rclee91 search PR goes in, we can trigger a Renovate PR for elasticsearch@v8 to see if it breaks.

[AmasiaNalbandian]

@DukeManh does that mean it will automatically do it? I forget how we are triggering them.

[DukeManh]

@AmasiaNalbandian, you can approve a major update to trigger the PR in Renovate dashboard here https://github.com/Seneca-CDOT/telescope/issues/2638

[humphd]

Please don't do this without testing. As I say, it's known to break our boxes.

[RC-Lee]

Reading the release notes, here are some parts I know will already break Search and tests

• Dropping callback API
• Removing body key from request
• The returned value of API calls is the body and not the http related keys

Some other things to note in the API changes (will have to scroll down all the way to Search for the whole list). I've listed some that is obvious to me.

• Fix range query on date fields for number inputs #63692 (issue: #63680)
• Remove type query #47207 (issue: #41059)

[manekenpix]

Some more info: I tried to run the parser with ES v8 (both image and client) and the first thing to deal with is this (the client can't connect to ES):

"log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection"

It seems that some security features are now enabled by default. In the docs, they mention this to disable security:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

So, what should we disable? I tested disabling the first option xpack.security.enabled: false, and the client can connect to ES (storing fails because of some of the issues mentioned in a previous comment). Disabling the second option (which presumably means the first option is enabled), causes this: xpack.security.transport.ssl.enabled: false

error: {
      "name": "ResponseError",
      "meta": {
        "body": {
          "error": {
            "root_cause": [
              {
                "type": "security_exception",
                "reason": "missing authentication credentials for REST request [/_cluster/health]",
                "header": {
                  "WWW-Authenticate": [
                    "Basic realm=\"security\" charset=\"UTF-8\"",
                    "ApiKey"
                  ]
                }
              }
            ]

My guess is that we can disable everything since we don't expose ES at all and supabse is going to take over soon. Any suggestions?

cc: @rclee91 @AmasiaNalbandian @humphd

[RC-Lee]

We're using ElasticSearch from Satellite in search and parser, so I'm guessing ES in Satellite will also have to be upgraded as well

[humphd]

I'm not sure that we'll dump ES any time soon, since Supabase doesn't provide the same capabilities. We might do that eventually, but not in 3.0.

I think that we should be able to avoid all the security to the ES instance because we don't expose it to the world. We'll have to figure out how to turn this stuff off, or how to satisfy it?

[humphd]

The eslint mock we use doesn't support 8.0 yet, so this is not happening until we get that. @rclee91 has details in https://github.com/Seneca-CDOT/telescope/pull/3026#issue-1147781142

[RC-Lee]

Unfortunate, but I think I'd be even more sad if we had to rip out the tests that I took so long to get in. Should I close that PR for now then?

[humphd]

Let's leave it open, but block it on these issues being sorted.

[AmasiaNalbandian]

I don't like this one at all.... :(

However, after seeing what @rclee91 detailed, this doesn't seem THAT bad (the documentation is really good for the migration). We can probably start a draft PR in 2.9 and prepare it for release 4.0 so we have a rough outline.

[humphd]

This can happen post-3.0