Discuss: Add support for multiple outbound contexts for CCS / CCR?

[justincr-elastic]

Background

Transport protocol settings do not support separate inbound vs outbound config.

1. Within a cluster, uniform trust is desirable for intra-cluster Transport protocol connections.
2. Between clusters, uniform trust for CCS/CCR is desirable, if all clusters are controlled by a single owner.
3. Between clusters, uniform trust for CCS/CCR is not desirable, if some clusters are controlled by different owners.

To support 3, separate outbound config is needed.

Discussion

Would it be sufficient to add a single outbound config, or are there use cases where multiple outbound contexts would be needed?

Transport Profiles example

Transport Profiles does allow config of different inbound contexts for CCS/CCR. However, there is no analogy for outbound context(s).

Using Profiles as an analogy, is a default outbound sufficient, or multiple outbounds?

# Inbound+outbound for local cluster1, and outbound to remote cluster2
transport.host: [ _local_ ] # localhost (127.0.0.1)
transport.port: 9311
xpack.security.transport.filter.enabled: true
xpack.security.transport.filter.allow: [ "127.0.0.0/8" ]
xpack.security.transport.filter.deny: _all
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.supported_protocols: [TLSv1.3]
xpack.security.transport.ssl.client_authentication: required # Options: required,optional,none
xpack.security.transport.ssl.verification_mode: certificate # Options: full,certificate,none
xpack.security.transport.ssl.certificate_authorities: [cluster1-trans-clientservers-internal-private-ca/ca.crt, cluster2-trans-clientservers-internal-private-ca/ca.crt]
xpack.security.transport.ssl.certificate: cluster1-trans-clientservers-internal-private-elasticsearch1/cluster1-trans-clientservers-internal-private-elasticsearch1.crt
xpack.security.transport.ssl.key: cluster1-trans-clientservers-internal-private-elasticsearch1/cluster1-trans-clientservers-internal-private-elasticsearch1.key

# Inbound from remote cluster2
#transport.profiles.default.xpack.security.filter.enabled: true
transport.profiles.default.xpack.security.filter.allow: [ "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16" ]
transport.profiles.default.xpack.security.filter.deny: _all
transport.profiles.inbound_from_cluster2.bind_host: [ _site_ ] # LAN (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)
transport.profiles.inbound_from_cluster2.port: 9411
transport.profiles.inbound_from_cluster2.xpack.security.ssl.supported_protocols: [TLSv1.3]
transport.profiles.inbound_from_cluster2.xpack.security.ssl.client_authentication: required # Options: required,optional,none
transport.profiles.inbound_from_cluster2.xpack.security.ssl.verification_mode: certificate # Options: full,certificate,none
transport.profiles.inbound_from_cluster2.xpack.security.ssl.certificate_authorities: [cluster2-trans-clientservers-internal-private-ca/ca.crt]
transport.profiles.inbound_from_cluster2.xpack.security.ssl.certificate: cluster1-trans-clientservers-internal-private-elasticsearch1/cluster1-trans-clientservers-internal-private-elasticsearch1.crt
transport.profiles.inbound_from_cluster2.xpack.security.ssl.key: cluster1-trans-clientservers-internal-private-elasticsearch1/cluster1-trans-clientservers-internal-private-elasticsearch1.key

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

1. I think the outbound config should be attached to the remote cluster definition, not a transport profile. Different remote clusters may want different trust configurations, and we may want to adjust these configurations at runtime in ways that are not really compatible with the transport profile model.

2. We may want to continue applying different trust configurations on inbound connections according to the port but I think this scales badly (the number of available ports is pretty limited in practice) and will be tricky to fit into Cloud. I think we should consider ways to share the port across multiple trust models by making the distinction at a higher protocol level using something like SNI. We may also want to adjust these inbound configurations at runtime in ways that are not really compatible with the transport profile model.

3. Inbound and outbound configs are qualitatively different for cross-cluster connections, and both are qualitatively different from intra-cluster connections. And transport profiles, which really originally existed for the transport client. I believe we should celebrate these differences rather than trying to hide them behind a uniform interface.

[justincr-elastic]

To clarify, I was thinking Transport Profiles for inbound, and something like a new xpack.security.transport.remotecluster.outbound.<contextname> for outbound.

Can Transport Profiles be reused for inbound? Maybe. It is desirable to support different ports, like Transport Profiles. However, if Transport Profiles cannot be reused, then new inbounds would be needed which duplicate most of what Transport Profiles is doing.

Consider this cluster:

• cluster1 <= cluster2 => cluster3

A mock up of what the remote cluster definitions on cluster2 might look like:

{
  "persistent" : {
    "cluster" : {
      "remote" : {
        "cluster1" : {
          "mode" : "remoteclustersniff",
          "seeds" : [ "es01.cluster1.example.com:9111", "es02.cluster1.example.com:9111" ]
        },
        "cluster3" : {
          "mode" : "remoteclustersniff",
          "seeds" : [ "es01.cluster3.example.com:9333", "es02.cluster3.example.com:9333" ],
          "xpack.security.transport.remotecluster.outbound" : "to_cluster3",
          "apikey" : "XXXXXXXXXXXXXXXXXXXXXXXX"
        }
      }
    }
  }
}

A property would map to an outbound context in YAML. Cluster1 would use default. Cluster3 would use the named to_cluster3 context. Note, the second example disables TLS client authentication, and will use the API Key from the remote cluster definition instead.

xpack.security.transport.remotecluster.outbound.default.ssl.client_authentication: required
xpack.security.transport.remotecluster.outbound.default.ssl.supported_protocols: [TLSv1.2, TLSv1.3]
xpack.security.transport.remotecluster.outbound.default.ssl.verification_mode: certificate
xpack.security.transport.remotecluster.outbound.default.ssl.certificate_authorities: [cluster1-trans-ca.crt]
xpack.security.transport.remotecluster.outbound.default.ssl.certificate: cluster2-trans-es01.crt
xpack.security.transport.remotecluster.outbound.default.ssl.key: cluster2-trans-es01.key

xpack.security.transport.remotecluster.outbound.to_cluster3.ssl.client_authentication: none
xpack.security.transport.remotecluster.outbound.to_cluster3.ssl.supported_protocols: [TLSv1.3]
xpack.security.transport.remotecluster.outbound.to_cluster3.ssl.verification_mode: full
xpack.security.transport.remotecluster.outbound.to_cluster3.ssl.certificate_authorities: [cluster3-trans-ca.crt]

@gwbrown and I came across additional examples of two larger CCS setups:

1. Central cluster does CCS to 65 remote clusters. All have the same owner. A single default outbound context would be sufficient, and config would scale nicely.
2. Central cluster does CCS to 100+ remote clusters. All have different owners. Clusters must adhere to zero trust, and CCS needs auditlogs on the remotes to log who did the remote search, and what did they query.

These are just 2 examples. I assume there would be other CCS examples using small, medium, or large numbers of remote clusters.

On the flip side, I assume there are use cases where many clusters do CCR into a single, remote cluster.

[DaveCTurner]

A property would map to an outbound context in YAML.

I wonder if that extra level of indirection is necessary? Why not define all the parameters on each remote connection directly? If they're defined in elasticsearch.yml then (a) they need to be kept in sync across multiple nodes and (b) they require a node restart to change.

Conversely you seem to be proposing putting sensitive things like API keys in the remote cluster definition. I'd expect things like that to be held in the keystore. That still has the problem of keeping them in sync across multiple nodes, but this is a general problem with sensitive settings for which we already have solutions, and at least the keystore can be reloaded dynamically.

[justincr-elastic]

API Keys are proposed to be in a filtered cluster setting.

[justincr-elastic]

Not planned.