Intentions are not working as expected with Cluster Peering setup

[mguegan]

Overview of the Issue

When using ACLs with default_policy = "deny", intentions are working as expected within the same cluster/datacenter. But when using Cluster Peering and trying to access exported services, I always succeed establish connections, and that with or without adding intentions (using 'allow' or 'deny' rules are not affecting the communication)...

Reproduction Steps

Using a Cluster Peering setup with 2 peer DCs, ACLs enabled and default_policy = "deny" on all agents :

1. Create a service A inside cluster A, and export it to cluster B
2. Create a service B with an upstream to service A (via local mesh gateways) inside cluster B
3. Observe that without any intention configured, communication is established

PS: I'm using Nomad

Consul info for both Client and Server

Client info

agent:
        check_monitors = 0
        check_ttls = 0
        checks = 9
        services = 10
build:
        prerelease = 
        revision = 009041f8
        version = 1.17.3
        version_metadata = 
consul:
        acl = enabled
        known_servers = 3
        server = false
runtime:
        arch = amd64
        cpu_count = 2
        goroutines = 268
        max_procs = 2
        os = linux
        version = go1.21.7
serf_lan:
        coordinate_resets = 0
        encrypted = true
        event_queue = 0
        event_time = 98
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 8017
        members = 5
        query_queue = 0
        query_time = 1

advertise_addr = "{{ GetInterfaceIP \"ens6\" }}"
client_addr    = "0.0.0.0"
data_dir       = "/var/lib/consul"
datacenter     = "legacy-dk2"
encrypt        = "XXX"
retry_join     = [ "..." ]
server         = false
acl = {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
  tokens {
    default = "..."
    agent = "..."
  }
}
auto_encrypt {
  tls = true
}
performance {
  raft_multiplier = 1
}
ports {
  grpc = 8502
}
tls {
  defaults {
    ca_file = "/etc/consul.d/tls/consul-agent-ca.pem"
    verify_incoming = true
    verify_outgoing = true
  }
  internal_rpc {
    verify_server_hostname = true
  }
}
telemetry {
  disable_hostname = true
  prometheus_retention_time = "60s"
}
ui_config {
  enabled = true
}

Server info

agent:                    
        check_monitors = 0
        check_ttls = 0 
        checks = 3    
        services = 3
build:                       
        prerelease =    
        revision = 009041f8
        version = 1.17.3
        version_metadata = 
consul:                 
        acl = enabled   
        bootstrap = false
        known_datacenters = 1
        leader = true
        leader_addr = ....:8300
        server = true 
raft:                  
        applied_index = 342664
        commit_index = 342664
        fsm_pending = 0
        last_contact = 0
        last_log_index = 342664
        last_log_term = 2757
        last_snapshot_index = 327770
        last_snapshot_term = 2590
        latest_configuration = [{Suffrage:Voter ID:...}]
        latest_configuration_index = 0
        num_peers = 2
        protocol_version = 3
        protocol_version_max = 3
        protocol_version_min = 0
        snapshot_version_max = 1
        snapshot_version_min = 0
        state = Leader
        term = 2757
runtime:
        arch = amd64
        cpu_count = 2
        goroutines = 353
        max_procs = 2
        os = linux
        version = go1.21.7
serf_lan:
        coordinate_resets = 0
        encrypted = true
        event_queue = 0
        event_time = 98
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 8017
        members = 5
        query_queue = 0
        query_time = 1
serf_wan:
        coordinate_resets = 0
        encrypted = true
        event_queue = 0
        event_time = 1
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 168
        members = 3
        query_queue = 0
        query_time = 1

advertise_addr   = "{{ GetInterfaceIP \"ens6\" }}"
bootstrap_expect = 3
client_addr      = "0.0.0.0"
data_dir         = "/var/lib/consul"
datacenter       = "legacy-dk2"
encrypt          = "..."
retry_join       = [ "... ]
server           = true
acl = {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
  tokens {
    default = "..."
    agent = "..."
  }
}
auto_encrypt {
  allow_tls = true
}
performance {
  raft_multiplier = 1
}
tls {
  defaults {
    ca_file = "/etc/consul.d/tls/consul-agent-ca.pem"
    cert_file = "/etc/consul.d/tls/consul-agent-server.pem"
    key_file = "/etc/consul.d/tls/consul-agent-server-key.pem"
    verify_incoming = true
    verify_outgoing = true
  }
  internal_rpc {
    verify_server_hostname = true
  }
}
telemetry {
  disable_hostname = true
  prometheus_retention_time = "60s"
}
ui_config {
  enabled = true
}

Operating system and Environment details

FCOS stable 39.20231204.3.3 Nomad v1.7.5