Remote cluster - API key security model - cluster privileges

[jakelandis]

This commit adds a new remote_cluster to the role and API keys to enable expressing cluster permissions across remote clusters similar to the remote_indices block for RCS 2.0 (API key based cross cluster security).

"remote_cluster": [
        {
            "privileges": [
                "monitor_enrich"
            ],
            "clusters": [
                "my_remote*"
            ]
        }
    ]

The motivation for this change to enable ES|QL executed over remote clusters to allow or disallow remote enrichment. As such, the only privilege allowed here is "monitor_enrich" which controls if users can use ES|QL ENRICH keyword as part of their queries for the remote cluster. RemoteClusterSecurityEsqlIT has the relevant functional tests since the only way to exercise this permission. This change is quite large since this change touches the role and role descriptors used for API keys. That requires updates and/or testing for CRUD role, API keys, file based roles, get user privileges (which echos the role), usage stats, and the various views of the role : role descriptors, roles (base/limited by), role intersections, and (effective) roles.

The data model design differs a bit from remote_indices in that it is a simpler data model which shares very little with the non-remote variant of cluster privileges. This allows the data to modeled only once via RemoteClusterPermissions and RemoteClusterPermissionGroup. RemoteClusterPermissions has N RemoteClusterPermissionGroup and is reused across the various vies of the roles. RemoteClusterPermissions is created early and is used as the primary contract with this new concept. Those data objects are self de/serializing and self toXcontent (we still use old school fromXContent and didn't want to mix and match old/new parsing strategies). Much like remote_indices, on transfer to the remote cluster, they will be converted into the non remote variant.

Also note that in this PR, the monitor_enrich privilege is explicitly auto added to the backing API key set of permissions for CCS. CCS_CLUSTER_PRIVILEGE_NAMES = { "cross_cluster_search", "monitor_enrich" }; ideally, we would group monitor_enrich inside of cross_cluster_search privilege and avoid needing to explicitly add the permission. However, due to some low level details w.r.t. how the has_privilege is used, this is required. I will follow up to see if we can change how has_privilege is called via ES|QL to avoid the need to explicitly add monitor_enrich to the backing API key (infavor of only having cross_cluster_search that encompases the needed privilege).

Also note, that today still need a local privilege of "cluster": ["cross_cluster_search"], (or more granular "cluster": ["monitor_enrich"] ) since ES|QL + ENRICH does not support remote only ENRICH. Also, ES|QL (with out enrich) over CCS requires the local privilege "indices": [ { "names": [""], "privileges": ["read_cross_cluster"] } ] since the local action that coordinates the workflow needs to execute locally.

Also, docs will come in a different PR.

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[elasticsearchmachine]

Hi @jakelandis, I've created a changelog YAML for you.

[slobodanadamovic]

Do we want to track the usage of roles with remote_cluster privileges like we do for remote_indices (here and here)?

[jakelandis]

we still need to address BWC concern when sending remote cluster permissions to a RCS 2.0 cluster on an older version.

Good catch. I have updated the code to be remote version aware.

• https://github.com/elastic/elasticsearch/pull/107493/commits/ed8fde7839a653b880d99d0d6cd66c6e6b6cdff4 -> plumbing needed to push the remote cluster version to check
• https://github.com/elastic/elasticsearch/pull/107493/commits/e01974a7e0c2c4aca57c6ae355b1cc8a8a1e3419 -> implement per version logic and unit test
• https://github.com/elastic/elasticsearch/pull/107493/commits/e1da2593f95c6d535c81c6e9cd0ca08e1fcba226 -> fix stuff i broke
• https://github.com/elastic/elasticsearch/pull/107493/commits/d6c4b6609986d810ba3755e0c3bb4f02ce4a2864 -> fix the backwards logic in the first attempt

I think this should warrant a new BWC test.

I don't think it does. I think a manual test is warranted, but this will likely change very infrequently, and the test would only be applicable one version back (due to when we GA RCS 2.0). There is not much nuance since it will work or it will not, due to we either send valid permissions or not, and the unit tests cover the logic for when we send permissions. There are some non-trivial costs in CPU, flakeyness, complexity, and time to write these BWC tests. In this case I think a quick spot check manually + the unit tests to ensure we send the write permission to the right version is sufficient.

[jakelandis]

Do we want to track the usage of roles with remote_cluster privileges like we do for remote_indices (here and here)?

Indeed. I already got the filestore, but missed the native store. Thanks! -> https://github.com/elastic/elasticsearch/pull/107493/commits/cdae9b85f2e5f1717a343060d59beaf7c952bc51

[jakelandis]

all 👍 code review nits included in https://github.com/elastic/elasticsearch/pull/107493/commits/9b7ab458a08b8bd682ae49bce15133e11a40cc92

[slobodanadamovic]

I think this should warrant a new BWC test.

I don't think it does. I think a manual test is warranted, but this will likely change very infrequently, and the test would only be applicable one version back (due to when we GA RCS 2.0). There is not much nuance since it will work or it will not, due to we either send valid permissions or not, and the unit tests cover the logic for when we send permissions. There are some non-trivial costs in CPU, flakeyness, complexity, and time to write these BWC tests. In this case I think a quick spot check manually + the unit tests to ensure we send the write permission to the right version is sufficient.

Good point. 👍