Setting ClientOptions.User causes kerberos realm to be left unset

[yuchengwu]

here is my code:

package main

import (
	"flag"
	"io/ioutil"
	"log"
	"strings"

	chdfs "github.com/colinmarc/hdfs/v2"        // v2.2.0
	krb "github.com/jcmturner/gokrb5/v8/client" // v8.4.1
	"github.com/jcmturner/gokrb5/v8/config"
	"github.com/jcmturner/gokrb5/v8/keytab"
)

func main() {
	username := flag.String("u", "", "user name")
	principal := flag.String("p", "", "principal")
	dtp := flag.String("d", "", "data transfer protection")
	realm := flag.String("r", "", "realm")
	keyTabFilePath := flag.String("t", "", "key tab file path")
	krb5CfgFilePaht := flag.String("c", "", "krb5 conf file path")
	nn := flag.String("n", "", "name node list")
	flag.Parse()

	opts := chdfs.ClientOptions{
		User:      *username,
		Addresses: strings.Split(*nn, ","),
	}
	if *keyTabFilePath != "" {
		// Load the keytab
		dat, err := ioutil.ReadFile(*keyTabFilePath)
		if err != nil {
			log.Fatalf("open %s file error: %v", *keyTabFilePath, err)
		}
		kt := keytab.New()
		if err := kt.Unmarshal(dat); err != nil {
			log.Fatalf("could not load client keytab: %v", err)
		}
		log.Printf("Keytab_file info:\n%s", kt.String())
		// Load the client krb5 config
		krb5Cfg, err := ioutil.ReadFile(*krb5CfgFilePaht)
		if err != nil {
			log.Fatalf("open %s file error: %v", *krb5CfgFilePaht, err)
		}
		conf, err := config.NewFromString(string(krb5Cfg))
		if err != nil {
			log.Fatalf("could not load krb5.conf %s: %v", string(krb5Cfg), err)
		}
		log.Printf("krb5_config file:\n%s", string(krb5Cfg))
		// Create the client with the keytab
		cl := krb.NewWithKeytab(*username, *realm, kt, conf)
		if err := cl.Login(); err != nil {
			log.Fatalf("login with user %s error: %v", *username, err)
		}
		opts.KerberosServicePrincipleName = *principal
		opts.KerberosClient = cl
		opts.DataTransferProtection = *dtp
	}

	c, err := chdfs.NewClient(opts)
	if err != nil {
		log.Fatalf("new hdfs client error: %v", err)
	}
	log.Printf("New client %v ok", c)
	defer func() {
		if err := c.Close(); err != nil {
			log.Fatalf("close conn %+v error: %v", opts, err)
		}
		log.Fatalf("Close client %v ok", c)
	}()
	if _, err := c.ReadDir("/"); err != nil {
		log.Printf("read / error: %v", err)
	}
	if err := c.CreateEmptyFile("/tmp/test123"); err != nil {
		log.Printf("create empry file /tmp/test123 error: %v", err)
	}
	if err := c.Remove("/tmp/test123"); err != nil {
		log.Printf("remove file /tmp/test123 error: %v", err)
	}
}

build and exectue the above code:

go build -o k k.go

./k -c ./krb5.conf -r TCE.COM -t ./hdfs_hadoop-client.keytab  -n hdfs-10-29-2-18.tcs.internal:9000,hdfs-10-29-2-152.tcs.internal:9000 -p hdfs/_HOST   -d integrity -u hdfs/hadoop-client

then got the following EOF error:

2022/03/23 17:53:22 read / error: open /: EOF
2022/03/23 17:53:22 create empry file /tmp/test123 error: create /tmp/test123: EOF                                                                                                                         ok
2022/03/23 17:53:22 remove file /tmp/test123 error: remove /tmp/test123: EOF

I have no idea what's wrong, it looks like I have been successfully authenticated & authorized, connected to namenodes, but can not access the dfs. The same code works fine for env without kerberos, any tips to help further debugging would be appreciate.

[colinmarc]

Hi @yuchengwu, thanks for the bug report.

What hadoop distro and version are you running? Can you look in the namenode logs and see what the error is there?

[yuchengwu]

hi @colinmarc thanks for the response, after looking at the namenode server logs, I found a mismatch exception (see below screen shot)

It looks like the expected target service principal is hdfs/[email protected], but we got hdfs/hadoop-client, then I digging around the code found that If we specified the username(that what I had done in my code) then kerberosRealm(see below at line 117, 96-105) would not be populated when initializing a new namenode connection,

https://github.com/colinmarc/hdfs/blob/b2bcdb85539bbe670509010e11ae8bd32b2db26f/internal/rpc/namenode.go#L96-L124

A empty value of NamenodeConnection.kerberosRealm will cause the user passed to namenodes without a realm, see below lines 335 - 337, note that the param kerberosRealm is passed by NamenodeConnection.kerberosRealm, that will eventually fails the handshake with nodenames.

https://github.com/colinmarc/hdfs/blob/b2bcdb85539bbe670509010e11ae8bd32b2db26f/internal/rpc/namenode.go#L334-L345

To fix, what I have done is to add an extra line to set the User to empty if kerberos is enabled, then everything works fine now,

the code I have changed:

[yuchengwu]

A misconfig issue and it has been resolved, so I'm closing it now.

[yuchengwu]

An advise: if User and KerberosClient are mutual exclusive fields for ClientOptions when kerberos is enabled, it's better to valid those fields before than we initializing the connection.

[colinmarc]

Thanks for figuring this out! This is definitely a bug.

[colinmarc]

Fixed in 1dee011.