kafka-cluster-sasl example: "Improper format of Kerberos configuration file while initializing krb5 library" error

[naja1s]

Following the Docker Compose: Setting Up a Three Node Confluent Platform Cluster with SASL process, encountering the error message "kadmin.local: Improper format of Kerberos configuration file while initializing krb5 library" at the first execution of the kadmin.local commands in step 4.

Also perhaps related... the path for the keytab files in those commands is /tmp/keytab/${principal}.keytab but they are later referenced as being in the /etc/kafka/secrets directory in thesecrets/*jaas.conf files, for example secrets/zookeeper_1_jaas.conf has /etc/kafka/secrets/zookeeper1.keytab. <<< EDIT: Disregard this point... I see that in the kerberos container, /tmp/keytab mounts to the secrets directory via docker-compose.yml line 74

Have performed a quick initial search for guidance and will continue to search for related kerberos config error.

Please advise if I've missed something. Thanks!

[naja1s]

$ docker-compose ps
           Name                Command     State   Ports
--------------------------------------------------------
kafkaclustersasl_kerberos_1   /config.sh   Up
$ docker-compose exec kerberos bash
# cat /etc/krb5.conf
[logging]
 default = FILE:/var/log/kerberos/krb5libs.log
 kdc = FILE:/var/log/kerberos/krb5kdc.log
 admin_server = FILE:/var/log/kerberos/kadmind.log

[libdefaults]
 default_realm = TEST.CONFLUENT.IO
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 # WARNING: We use weaker key types to simplify testing as stronger key types
 # require the enhanced security JCE policy file to be installed. You should
 # NOT run with this configuration in production or any real environment. You
 # have been warned.
 default_tkt_enctypes = des-cbc-md5 des-cbc-crc des3-cbc-sha1
 default_tgs_enctypes = des-cbc-md5 des-cbc-crc des3-cbc-sha1
 permitted_enctypes = des-cbc-md5 des-cbc-crc des3-cbc-sha1

[realms]
 TEST.CONFLUENT.IO = {
  kdc =
  admin_server =
 }

[domain_realm]
 .test.confluent.io = TEST.CONFLUENT.IO
 test.confluent.io = TEST.CONFLUENT.IO

[naja1s]

Nothing in /var/log/kerberos/ in docker container:

# find /var/log
/var/log
/var/log/btmp
/var/log/tallylog
/var/log/wtmp
/var/log/ConsoleKit
/var/log/lastlog
/var/log/yum.log
/var/log/kerberos

krb5kdc isn't running and attempts to start produce the same error:

# /etc/rc.d/init.d/krb5kdc status
krb5kdc is stopped
# /etc/rc.d/init.d/krb5kdc start
Error initializing Kerberos: Improper format of Kerberos configuration file.
Starting Kerberos 5 KDC: krb5kdc: Improper format of Kerberos configuration file while initializing krb5
                                                           [FAILED]

[naja1s]

# cat /var/kerberos/krb5kdc/kdc.conf
[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

[realms]
 TEST.CONFLUENT.IO = {
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  # WARNING: We use weaker key types to simplify testing as stronger key types
  # require the enhanced security JCE policy file to be installed. You should
  # NOT run with this configuration in production or any real environment. You
  # have been warned.
  master_key_type = des3-hmac-sha1
  supported_enctypes = arcfour-hmac:normal des3-hmac-sha1:normal des-cbc-crc:normal des:normal des:v4 des:norealm des:onlyrealm des:afs3
  default_principal_flags = +preauth
 }
# cat /var/kerberos/krb5kdc/kadm5.acl
*/[email protected] *

[naja1s]

I believe step 3 has an error too. It says that the docker-compose commands are to be ran after ensuring "you are in the cp-docker-images directory." Shouldn't it instruct to be in the cp-docker-images/examples/kafka-cluster-sasl directory, where the docker-compose.yml file is?

[naja1s]

Also getting the same result if attempting steps 4 and 5 of the Docker Client: Setting Up a Three Node Kafka Cluster process.

[naja1s]

Found that Kerberos source has a Python-based configuration file validator. Used it in the manner shown below to validate both /etc/krb5.conf and /var/kerberos/krb5kdc/kdc.conf and discovered:

• /etc/krb5.conf
  • des-cbc-md5 and des-cbc-crc seem not to be valid values for either libdefaults.default_tkt_enctypes or libdefaults.default_tgs_enctypes
• /var/kerberos/krb5kdc/kdc.conf
  • arcfour-hmac:normal, des-cbc-crc:normal, des:normal, des:v4, des:norealm, des:onlyrealm, and des:afs3 do not seem to be valid values for realms.TEST.CONFLUENT.IO.supported_enctypes
  • admin_keytab is not a recognized attribute for realms.TEST.CONFLUENT.IO

$ export KAFKA_SASL_SECRETS_DIR=$(pwd)/secrets
$ docker run -d --name=kerberos --net=host -v ${KAFKA_SASL_SECRETS_DIR}:/tmp/keytab -v /dev/urandom:/dev/random confluentinc/cp-kerberos
...
$ docker exec -it kerberos bash
# yum install -y tar PyYAML
...
# curl https://kerberos.org/dist/krb5/1.16/krb5-1.16.tar.gz >krb5-1.16.tar.gz
# tar xzf krb5-1.16.tar.gz
# cd krb5-1.16/src/util/confvalidator
# python validator.py /etc/krb5.conf -c validator.conf
Wrong type des-cbc-md5 for attribute root.libdefaults.default_tkt_enctypes
Wrong type des-cbc-crc for attribute root.libdefaults.default_tkt_enctypes
Wrong type des-cbc-md5 for attribute root.libdefaults.default_tgs_enctypes
Wrong type des-cbc-crc for attribute root.libdefaults.default_tgs_enctypes
# python validator.py /var/kerberos/krb5kdc/kdc.conf -c validator.conf
Wrong type arcfour-hmac:normal for attribute root.realms.TEST.CONFLUENT.IO.supported_enctypes
Wrong type des-cbc-crc:normal for attribute root.realms.TEST.CONFLUENT.IO.supported_enctypes
Wrong type des:normal for attribute root.realms.TEST.CONFLUENT.IO.supported_enctypes
Wrong type des:v4 for attribute root.realms.TEST.CONFLUENT.IO.supported_enctypes
Wrong type des:norealm for attribute root.realms.TEST.CONFLUENT.IO.supported_enctypes
Wrong type des:onlyrealm for attribute root.realms.TEST.CONFLUENT.IO.supported_enctypes
Wrong type des:afs3 for attribute root.realms.TEST.CONFLUENT.IO.supported_enctypes
Unrecognized attribute admin_keytab at root.realms.TEST.CONFLUENT.IO

[naja1s]

Continuing from my previous post... I've edited the configuration files, and getting good (no?) output from the validator, but still observing a failure to start krb5kdc due to same original error.

# cat /etc/krb5.conf
[logging]
 default = FILE:/var/log/kerberos/krb5libs.log
 kdc = FILE:/var/log/kerberos/krb5kdc.log
 admin_server = FILE:/var/log/kerberos/kadmind.log

[libdefaults]
 default_realm = TEST.CONFLUENT.IO
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 # WARNING: We use weaker key types to simplify testing as stronger key types
 # require the enhanced security JCE policy file to be installed. You should
 # NOT run with this configuration in production or any real environment. You
 # have been warned.
 default_tkt_enctypes = des3-cbc-sha1
 default_tgs_enctypes = des3-cbc-sha1
 permitted_enctypes = des-cbc-md5 des-cbc-crc des3-cbc-sha1

[realms]
 TEST.CONFLUENT.IO = {
  kdc =
  admin_server =
 }

[domain_realm]
 .test.confluent.io = TEST.CONFLUENT.IO
 test.confluent.io = TEST.CONFLUENT.IO
# cat /var/kerberos/krb5kdc/kdc.conf
[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

[realms]
 TEST.CONFLUENT.IO = {
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  # WARNING: We use weaker key types to simplify testing as stronger key types
  # require the enhanced security JCE policy file to be installed. You should
  # NOT run with this configuration in production or any real environment. You
  # have been warned.
  master_key_type = des3-hmac-sha1
  supported_enctypes = des3-hmac-sha1:normal
  default_principal_flags = +preauth
 }
# python validator.py /etc/krb5.conf -c validator.conf
# echo $?
0
# python validator.py /var/kerberos/krb5kdc/kdc.conf -c validator.conf
# echo $?
0
# service krb5kdc start
Error initializing Kerberos: Improper format of Kerberos configuration file.
Starting Kerberos 5 KDC: krb5kdc: Improper format of Kerberos configuration file while initializing krb5
                                                           [FAILED]

[lesaux]

I saw this today as well and added --hostname=test.confluent.io to the docker run command.

docker run --name=kerberos --hostname=test.confluent.io --net=host   -v ${KAFKA_SASL_SECRETS_DIR}:/tmp/keytab   -v /dev/urandom:/dev/random   confluentinc/cp-kerberos:5.0.0
+ : TEST.CONFLUENT.IO
+ : test.confluent.io
+ : masterkey
+ : admin
+ : admin
+ [[ /config.sh == \/\c\o\n\f\i\g\.\s\h ]]
+ main
+ '[' '!' -f /kerberos_initialized ']'
+ create_config
++ hostname -f
+ : confluentio.wpengine.com
+ cat
+ cat
+ create_db
+ /usr/sbin/kdb5_util -P masterkey -r TEST.CONFLUENT.IO create -s
Couldn't open log file /var/log/kerberos/kadmind.log: No such file or directory
Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'TEST.CONFLUENT.IO',
master key name 'K/[email protected]'
+ create_admin_user
+ kadmin.local -q 'addprinc -pw admin admin/admin'
Couldn't open log file /var/log/kerberos/kadmind.log: No such file or directory
WARNING: no policy specified for admin/[email protected]; defaulting to no policy
Authenticating as principal root/[email protected] with password.
Principal "admin/[email protected]" created.
+ echo '*/[email protected] *'
+ start_kdc
+ mkdir -p /var/log/kerberos
+ /etc/rc.d/init.d/krb5kdc start
Starting Kerberos 5 KDC: [  OK  ]
+ /etc/rc.d/init.d/kadmin start
Starting Kerberos 5 Admin Server: [  OK  ]
+ chkconfig krb5kdc on
+ chkconfig kadmin on
+ touch /kerberos_initialized
+ '[' '!' -f /var/kerberos/krb5kdc/principal ']'
+ start_kdc
+ mkdir -p /var/log/kerberos
+ /etc/rc.d/init.d/krb5kdc start
Starting Kerberos 5 KDC:
+ /etc/rc.d/init.d/kadmin start
Starting Kerberos 5 Admin Server:
+ chkconfig krb5kdc on
+ chkconfig kadmin on
+ tail -F /var/log/kerberos/krb5kdc.log
Oct 25 16:01:38 test.confluent.io krb5kdc[21](info): listening on fd 9: udp fe80::42:85ff:fe03:ab2d%docker0.88
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 25 16:01:38 test.confluent.io krb5kdc[21](info): listening on fd 10: udp fe80::42:daff:fe8c:2d59%br-c51fc368a37e.88
krb5kdc: setsockopt(11,IPV6_V6ONLY,1) worked
Oct 25 16:01:38 test.confluent.io krb5kdc[21](info): listening on fd 11: udp fe80::42:e3ff:fe2b:2327%br-fa68da9bc362.88
krb5kdc: setsockopt(12,IPV6_V6ONLY,1) worked
Oct 25 16:01:38 test.confluent.io krb5kdc[21](info): listening on fd 13: tcp 0.0.0.0.88
Oct 25 16:01:38 test.confluent.io krb5kdc[21](info): listening on fd 12: tcp ::.88
Oct 25 16:01:38 test.confluent.io krb5kdc[21](info): set up 7 sockets
Oct 25 16:01:38 test.confluent.io krb5kdc[22](info): commencing operation
tail: unrecognized file system type 0x794c7630 for `/var/log/kerberos/krb5kdc.log'. Reverting to polling.

[jocelyndrean]

@naja1s Also to fix this issue, please consider to carefully follow theses steps :

You must create an entry in /etc/hosts with hostname quickstart.confluent.io that points to eth0 IP. In Linux, run the below commands on the Linux host.

If running Docker Machine (eg for Mac or Windows), you will need to SSH into the VM and run the below commands as root. You can SSH into the Docker Machine VM by running docker-machine ssh confluent.

export ETH0_IP=$(ifconfig eth0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}')

echo ${ETH0_IP} quickstart.confluent.io >> /etc/hosts

And then you can build and run your kerberos image:

cd tests/images/kerberos
docker build -t confluentinc/cp-kerberos:4.1.3 .

docker run -d \
  --name=kerberos \
  --net=host \
  -v ${KAFKA_SASL_SECRETS_DIR}:/tmp/keytab \
  -v /dev/urandom:/dev/random \
  confluentinc/cp-kerberos:4.1.3

Please, also consider to close this issue if it's fixed :)

[mroiter-larus]

Hi, i'm having the same issue as @naja1s. I tried all the possible fixes provided in the above comments but without success. Everything is ok until i start to create all principals and keytabs (step 4.2 of the Docker Compose: Setting Up a Three Node Confluent Platform Cluster with SASL section). I got the following error:

kadmin.local: Improper format of Kerberos configuration file while initializing krb5 library

Any suggestion please?

[mroiter-larus]

I tried also the approach suggested into Docker Client: Setting Up a Three Node Kafka Cluster section. In this case i'm able to generate the principals for zookeepers, zkclients, brokers, saslconsumer and saslproducer. After running the 3 Zookeeper containers, i ran the docker logs zk-sasl-1 command and i got the following error:

KDCCommunication: kdc=quickstart.confluent.io UDP:88, timeout=30000,Attempt =1, #bytes=178 SocketTimeOutException with attempt: 1 KDCCommunication: kdc=quickstart.confluent.io UDP:88, timeout=30000,Attempt =2, #bytes=178 SocketTimeOutException with attempt: 2 KDCCommunication: kdc=quickstart.confluent.io UDP:88, timeout=30000,Attempt =3, #bytes=178 SocketTimeOutException with attempt: 3 KrbKdcReq send: error trying quickstart.confluent.io java.net.SocketTimeoutException: Receive timed out at java.net.PlainDatagramSocketImpl.receive0(Native Method) at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:143) at java.net.DatagramSocket.receive(DatagramSocket.java:812) at sun.security.krb5.internal.UDPClient.receive(NetClient.java:206) at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:411) at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:364) at java.security.AccessController.doPrivileged(Native Method) at sun.security.krb5.KdcComm.send(KdcComm.java:348) at sun.security.krb5.KdcComm.sendIfPossible(KdcComm.java:253) at sun.security.krb5.KdcComm.send(KdcComm.java:229) at sun.security.krb5.KdcComm.send(KdcComm.java:200) at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:316) at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361) at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776) at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at javax.security.auth.login.LoginContext.login(LoginContext.java:587) at org.apache.zookeeper.Login.login(Login.java:294) at org.apache.zookeeper.Login.(Login.java:94) at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:208) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:130) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) KdcAccessibility: add quickstart.confluent.io [2020-02-20 17:11:16,129] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain) java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Receive timed out at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:130) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)

My krb5.conf file, into the kerberos container, is the following:

[logging] default = FILE:/var/log/kerberos/krb5libs.log kdc = FILE:/var/log/kerberos/krb5kdc.log admin_server = FILE:/var/log/kerberos/kadmind.log

[libdefaults] default_realm = TEST.CONFLUENT.IO dns_lookup_realm = false dns_lookup_kdc = false ticket_lifetime = 24h renew_lifetime = 7d forwardable = true

default_tkt_enctypes = des3-cbc-sha1 default_tgs_enctypes = des3-cbc-sha1 permitted_enctypes = des-cbc-md5 des-cbc-crc des3-cbc-sha1

[realms] TEST.CONFLUENT.IO = { kdc = quickstart.confluent.io admin_server = quickstart.confluent.io }

[domain_realm] .test.confluent.io = TEST.CONFLUENT.IO test.confluent.io = TEST.CONFLUENT.IO

My kdc.conf file, into the kerberos container, is the following:

[kdcdefaults] kdc_ports = 88 kdc_tcp_ports = 88

[realms] TEST.CONFLUENT.IO = { acl_file = /var/kerberos/krb5kdc/kadm5.acl dict_file = /usr/share/dict/words master_key_type = des3-hmac-sha1 supported_enctypes = des3-hmac-sha1:normal default_principal_flags = +preauth }

Please note that i've already try to change [realms] configuration in my krb5.conf as follow:

--This was the original configuration-- [realms] TEST.CONFLUENT.IO = { kdc = docker-desktop admin_server = docker-desktop }

and also

--This was the configuration generated using confluentinc/cp-kerberos:latest instead of 4.1.0-- [realms] TEST.CONFLUENT.IO = { kdc = admin_server = }

In both cases i got the same error running the Zookeeper container.

I hope you can help me understand where the problem is.

Thanks in advance!

[mroiter-larus]

According to #468 i solved my issue downgrading to confluentinc/cp-kafka:4.0.0 for my brokers.