webhdfs icon indicating copy to clipboard operation
webhdfs copied to clipboard

Adds Knox Support

Open att-bdpaas opened this issue 10 years ago • 2 comments

We are using WebHDFS through a Knox gateway.

This requires:

  1. Specifying the gateway and the cluster to the path
  2. Support for basic auth

Let us know if you have any questions/ feedback about this PR.

Thanks!

att-bdpaas avatar Nov 18 '14 00:11 att-bdpaas

I'm sorry for late reply. Are there any other Knox gateway users? I cannot find anyone around me. I'll re-consider to merge if any others adds comments for merge.

About patch, I think that we should change API of initialize like below, and change username/doas/proxy_address/proxy_port as deprecated:

def initialize(host='localhost', port=50070, username=nil, doas=nil, proxy_address=nil, proxy_port=nil, options={})
   # check options[:username], options[:doas], ... , options[:knox_path], ...
   # ...
end

tagomoris avatar Dec 01 '14 07:12 tagomoris

Knox is obviously relatively new, but is definitely shaping up to be the standard way to connect to a secure Kerberized cluster. Both Hortonworks and Cloudera now have it available.

Regarding the initialize method this seems like a decent approach. Though it does seem arbitrary what goes in options and what goes as positioned arguments. Ruby's 2.0 named arguments would solve this but guessing you want your library to support as many Ruby versions as possible.

tlabeeuw avatar Dec 01 '14 18:12 tlabeeuw