allow to configure dns_record_type with a system property

[cpesch]

My setup is a set of Keycloak 18.0.2 clusters in an environment with nomad and consul. We're using DNS_PING with SRV records for a long time now and the JGroups setup is pretty simple:

JGROUPS_DISCOVERY_PROTOCOL=dns.DNS_PING
JGROUPS_DISCOVERY_PROPERTIES="dns_query="${dns_query}",dns_record_type="SRV""
JGROUPS_TRANSPORT_STACK="tcp"

When migrating this to Keycloak 23.0.7, I found that the dns_record_type separated with a comma is not supported anymore. With the documentation I found on the internet and discussions and issue in the JGroups, Infinispan, Keycloak github projects, I failed to get SRV records running with something straight-forward like this:

<infinispan ...>

    <jgroups>
        <stack name="dns-ping">
            <dns.DNS_PING dns_record_type="SRV"
                          dns_query="${dns_query}"/>
            <TCP diag.enabled="true"
                 bind_addr="GLOBAL"
                 external_addr="{{ env "NOMAD_IP_jgroupscluster" }}"
                 external_port="{{ env "NOMAD_HOST_PORT_jgroupscluster" }}"/>
        </stack>
    </jgroups>

    <cache-container name="keycloak">
        <transport stack="dns-ping"...

When looking at org.jgroups.protocols.dns.DNS_PING I found that only a tiny tweak is necessary to reduce all this down to

KC_CACHE=ispn
KC_CACHE_STACK=kubernetes
JAVA_OPTS_APPEND="-Djgroups.dns.dns_query=${dns_query} -Djgroups.dns.dns_record_type=SRV

This works fine for me since some days.

@belaban I'd be really happy if you include this in an upcoming 5.2.x release

[belaban]

Hmm... adding the system property should not be required, although it certainly works, so no objections. This could be done in the XML config directly: JGroups parses comma-deparated attributes and tries to substitute both system properties and environment variables.

[belaban]

In other words, no change and no release should be required...

[rhusar]

Hmm... adding the system property should not be required, although it certainly works, so no objections. This could be done in the XML config directly: JGroups parses comma-deparated attributes and tries to substitute both system properties and environment variables.

IIUC well, no - the whole point is so that you do not have to edit the XML as that's way more convoluted in a container rather than adding a property (this PR) and then simply adding -Djgroups.dns.dns_record_type=SRV to the start options.

[belaban]

Christian is already using a system prop (JGROUPS_DISCOVERY_PROPERTIES) to change the config, so it could be done there (IMO)... BTW: a JIRA would be nice, so we can assign this change to a version

[cpesch]

IIUC well, no - the whole point is so that you do not have to edit the XML as that's way more convoluted in a container rather than adding a property (this PR) and then simply adding -Djgroups.dns.dns_record_type=SRV to the start option

@rhusar well said. I fully agree.

Christian is already using a system prop (JGROUPS_DISCOVERY_PROPERTIES) to change the config, so it could be done there (IMO)...

I'm confused: that was the Keycloak 18 legacy configuration.

Wait. Is there a way to set org.jgroups.protocols.dns.DNS_PING#dns_record_type without touching the XML? Just with a system property or environment variable?

[belaban]

Yes, in the config (XML or programmatic):

<dns.DNS dns_record_type="${dns.record_type,RECORD_TYPE:A}".../>

Default is A. System property dns.record_type or env var RECORD_TYPE overrird this.

[cpesch]

Ok, that's exactly the config I was struggeling with. I spent two days reading and experimenting – and couldn't even see this in the logs:

2024-03-07 11:36:59,929 TRACE [org.jgroups.protocols.dns.DefaultDNSResolver] (jgroups-13,040f8502066a-48384) resolving DNS query: labs-keycloak-playground..... of a type: SRV

It might be caused by JGroups being integrated to Infinispan, and Infinispan being integrated in Keycloak. I've tried to find people who were successful with this. Could you spot, what's wrong here in my config?

<infinispan
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:14.0 http://www.infinispan.org/schemas/infinispan-config-14.0.xsd"
    xmlns="urn:infinispan:config:14.0">

<jgroups>
    <stack name="dns-ping">
        <dns.DNS_PING dns_record_type="SRV"
                      dns_query="${dns_query}"
                      stack.combine="REPLACE"
                      stack.position="MPING"/>
        <TCP diag.enabled="true"
             bind_addr="GLOBAL"
             external_addr="{{ env "NOMAD_IP_jgroupscluster" }}"
             external_port="{{ env "NOMAD_HOST_PORT_jgroupscluster" }}"/>
    </stack>
</jgroups>

<cache-container name="keycloak">
    <transport lock-timeout="60000" stack="dns-ping"/>
    <local-cache name="realms" simple-cache="true">
        <encoding>
            <key media-type="application/x-java-object"/>
            <value media-type="application/x-java-object"/>
        </encoding>
        <memory max-count="10000"/>
    </local-cache>
    <local-cache name="users" simple-cache="true">
        <encoding>
            <key media-type="application/x-java-object"/>
            <value media-type="application/x-java-object"/>
        </encoding>
        <memory max-count="10000"/>
    </local-cache>
    <distributed-cache name="sessions" owners="2">
        <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="authenticationSessions" owners="2">
        <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="offlineSessions" owners="2">
        <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="clientSessions" owners="2">
        <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="offlineClientSessions" owners="2">
        <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="loginFailures" owners="2">
        <expiration lifespan="-1"/>
    </distributed-cache>
    <local-cache name="authorization" simple-cache="true">
        <encoding>
            <key media-type="application/x-java-object"/>
            <value media-type="application/x-java-object"/>
        </encoding>
        <memory max-count="10000"/>
    </local-cache>
    <replicated-cache name="work">
        <expiration lifespan="-1"/>
    </replicated-cache>
    <local-cache name="keys" simple-cache="true">
        <encoding>
            <key media-type="application/x-java-object"/>
            <value media-type="application/x-java-object"/>
        </encoding>
        <expiration max-idle="3600000"/>
        <memory max-count="1000"/>
    </local-cache>
    <distributed-cache name="actionTokens" owners="2">
        <encoding>
            <key media-type="application/x-java-object"/>
            <value media-type="application/x-java-object"/>
        </encoding>
        <expiration max-idle="-1" lifespan="-1" interval="300000"/>
        <memory max-count="-1"/>
    </distributed-cache>
</cache-container>
</infinispan>

It looks simple but with this config, I don't see any output from the org.jgroups.protocols.dns.DefaultDNSResolver and I've tried a lot of variants. What am I missing?

[pruivo]

You need to extend the kubernetes stack:

      <stack name="dns-ping" extends="kubernetes">
         <dns.DNS_PING dns_record_type="SRV"/>
      </stack>

[cpesch]

Finally, I've found the road blocker for my efforts: the distributed cache configuration in Keycloak is part of the build time configuration and not the run time configuration. After putting this cache configuration file

<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:14.0 http://www.infinispan.org/schemas/infinispan-config-14.0.xsd"
        xmlns="urn:infinispan:config:14.0">

    <jgroups>
        <stack name="dns-ping" extends="kubernetes">
            <dns.DNS_PING dns_query="${jgroups.dns.query}"
                          dns_record_type="${jgroups.dns.record:A}"
            />
        </stack>
    </jgroups>

    <cache-container name="keycloak">
        <transport lock-timeout="60000" stack="dns-ping"/>
...same as above...

into the docker image, running kc.sh build, I could configure it with

JAVA_OPTS_APPEND="-Xmx${kc_xmx}m -Djgroups.bind_addr=GLOBAL -Djgroups.external_addr={{ env "NOMAD_IP_jgroupscluster" }} -Djgroups.external_port={{ env "NOMAD_HOST_PORT_jgroupscluster" }} -Djgroups.dns.query=${dns_query} -Djgroups.dns.record=SRV"

at run time like for Keycloak 18 – and it works. This was the discussion that finally helped me.

[belaban]

Sorry to let this one slip! Is this issue still relevant? Or would a change from JGroups' side be needed to make this less complicated?

[cpesch]

@belaban the change would remove some hours of researching and creating a custom XML configuration file in Keycloak/Infinispan to use SRV records for DNS lookup. IMHO it's still relevant as this is small change from JGroups' side would make this less complicated