elastic
Add explicit version check against ES version on startup
Problem Description
On startup the framework should check the ES version that it's running against.
Proposed Solution
If there is a mismatch we can handle it in a similar way as Kibana does it:
| Situation | Example Conenctor Framework version | Example ES version | Outcome |
|---|---|---|---|
| Versions are the same. | 7.15.1 | 7.15.1 | 💚 OK |
| ES patch number is newer. | 7.15.0 | 7.15.1 | ⚠️ Logged warning |
| ES minor number is newer. | 7.14.2 | 7.15.0 | ⚠️ Logged warning |
| ES major number is newer. | 7.15.1 | 8.0.0 | 🚫 Fatal error |
| ES patch number is older. | 7.15.1 | 7.15.0 | ⚠️ Logged warning |
| ES minor number is older. | 7.15.1 | 7.14.2 | 🚫 Fatal error |
| ES major number is older. | 8.0.0 | 7.15.1 | 🚫 Fatal error |
Alternatives
Right now it seems that we support running against all ES versions, any reasons to keep doing it?
Putting as medium effort as we need to think if proposed version compatibiltiy matrix makes sense
I love this issue to add compatibility checks ❤️ Have we considered also adding a periodic runtime check (not only at startup)? For example, what if the user upgraded Elasticsearch while the Connector Service is running?
I love this issue to add compatibility checks ❤️ Have we considered also adding a periodic runtime check (not only at startup)? For example, what if the user upgraded Elasticsearch while the Connector Service is running?
One issue that came up during the team discussion is that it might work fine for stack, we should be version-agnostic for the ES serverless that doesn't advertise any "version".
+1 for Serverless, connectors should always be compatible regardless of the version of connectors and the version of Elasticsearch running on Serverless.
Putting as medium effort as we need to think if proposed version compatibiltiy matrix makes sense
@jedrazb do you remember if this was discussed already? If so, is the current matrix up-to-date?
@navarone-feekery it was discussed that while such matrix can work well for stack, it will likely be problematic with serverless. If we can enforce in just for stack it should be fine! Yeah matrix is stolen from Kibana 😃 so it should be up-to-date.
It seems we can differentiate serverless / not serverless using root API GET / against the ES instance. The value is under version.build_flavor. We can also use version.number for the stack version.
Response from serverless
{
"name": "serverless",
"cluster_name": "fbc4a808c213432f8a351a756bada344",
"cluster_uuid": "CiUbgt0JSQ-6yj-kzpqRMw",
"version": {
"number": "8.11.0",
"build_flavor": "serverless",
"build_type": "docker",
"build_hash": "00000000",
"build_date": "2023-10-31",
"build_snapshot": false,
"lucene_version": "9.7.0",
"minimum_wire_compatibility_version": "8.11.0",
"minimum_index_compatibility_version": "8.11.0"
},
"tagline": "You Know, for Search"
}
Response from cloud
{
"name": "instance-0000000000",
"cluster_name": "ffb1ec38e38c454d9f26e50b276a725d",
"cluster_uuid": "Udj-6JxTT2WK7I9s-t8Ewg",
"version": {
"number": "8.14.0",
"build_flavor": "default",
"build_type": "docker",
"build_hash": "77a23e5d9fcb1ae2a56820e51b49b85ab5ebafb6",
"build_date": "2024-04-18T10:05:33.199326758Z",
"build_snapshot": false,
"lucene_version": "9.10.0",
"minimum_wire_compatibility_version": "7.17.0",
"minimum_index_compatibility_version": "7.0.0"
},
"tagline": "You Know, for Search"
}
+1, for Serverless, we need to use version.build_flavor for identification because Serverless has no notion of versioning and all Serverless projects will show a static 8.11.0 value in version.number (it does not mean it's not upgrading behind the scenes).