SASL handshake: bytes shorter than header length

[jsanko9]

Hello

I'm getting following error on our Kerberos secured cluster. "Couldn't connect to namenode: no available namenodes: SASL handshake: bytes shorter than header length"

Its semi-old cluster in enterprise environment running 3.0.0-cdh6.3.3

Here is extract of core-site.xml:

<configuration>
  <property>
    <name>io.compression.codecs</name> 
    <value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compre 
ss.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.Lz4Codec</value>
  </property>
  <property>
    <name>hadoop.security.authorization</name>
    <value>true</value>
  </property>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://server.subset.domain.com:8020</value>
  </property>
  <property>
    <name>hadoop.security.authentication</name>
    <value>kerberos</value>
  </property>
  <property>
    <name>hadoop.security.auth_to_local</name>
    <value>RULE:[1:$1@$0](.*@\QREGION.COMPANY.COM\E$)s/@\QREGION.COMPANY.COM\E$//
RULE:[2:$1@$0](.*@\QREGION.COMPANY.COM\E$)s/@\QREGION.COMPANY.COM\E$//
RULE:[1:$1@$0](.*@\QREGION.COMPANY.COM\E$)s/@\QREGION.COMPANY.COM\E$//
RULE:[2:$1@$0](.*@\QREGION.COMPANY.COM\E$)s/@\QREGION.COMPANY.COM\E$//
RULE:[1:$1@$0](.*@\QREGION.COMPANY.COM\E$)s/@\QREGION.COMPANY.COM\E$//
RULE:[2:$1@$0](.*@\QREGION.COMPANY.COM\E$)s/@\QREGION.COMPANY.COM\E$//
RULE:[1:$1@$0](.*@\QREGION.COMPANY.COM\E$)s/@\QREGION.COMPANY.COM\E$//
RULE:[2:$1@$0](.*@\QREGION.COMPANY.COM\E$)s/@\QREGION.COMPANY.COM\E$//
RULE:[1:$1@$0](.*@\QREGION.COMPANY.COM\E$)s/@\QREGION.COMPANY.COM\E$//
RULE:[2:$1@$0](.*@\QREGION.COMPANY.COM\E$)s/@\QREGION.COMPANY.COM\E$//
DEFAULT
DEFAULT</value>
  </property>

Biggest problem is that I don't know what to try next. Kinit is working fine, I tried to install old tested OpenSSL , I have tried multiple versions of debian and versions of hdfs go client. Even tried to play a little with sources, I managed to check that I'm actually getting 0 bytes back.

Any suggestions ?

[jsanko9]

Not sure if helpful at all but if I change: dfs.namenode.kerberos.principal hdfs/[email protected]

to "host/{{actual hostname}}@REGION.COMPANY.COM" I get:

Couldn't connect to namenode: no available namenodes: SASL handshake: wrong Token ID. Expected 0504, was 607f

not sure if that's better or worse ( I get original error with host/[email protected] )