mkdir is failing with the following error

[nkurn]

2021-03-03 23:27:05,940 INFO fs.ObjectStoreVisitor: Stocator registered as cos for cos://hdfsVault.mycos/dir1 2021-03-03 23:27:05,952 INFO cos.COSAPIClient: Init : cos://hdfsVault.mycos/dir1 2021-03-03 23:27:07,195 WARN cos.COSAPIClient: file status dir1 returned 404 -mkdir: Fatal internal error java.lang.NullPointerException at com.ibm.stocator.fs.common.ObjectStoreGlobber.glob(ObjectStoreGlobber.java:180) at com.ibm.stocator.fs.ObjectStoreFileSystem.globStatus(ObjectStoreFileSystem.java:544) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:353) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:250) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:233) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:104) at org.apache.hadoop.fs.shell.Command.run(Command.java:177) at org.apache.hadoop.fs.FsShell.run(FsShell.java:327) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.FsShell.main(FsShell.java:390)

my core-site.xml

fs.stocator.scheme.list cos fs.cos.impl com.ibm.stocator.fs.ObjectStoreFileSystem fs.stocator.cos.impl com.ibm.stocator.fs.cos.COSAPIClient fs.stocator.cos.scheme cos fs.cos.mycos.v2.signer.type false fs.cos.mycos.endpoint xx fs.cos.mycos.access.key xx fs.cos.mycos.secret.key xx

[nkurn]

has anyone seen this problem?

thanks.

[nkurn]

Hi This is the command I used

hdfs dfs -mkdir cos://hdfsVault.applogs/1/

2021-03-26 17:00:29,873 INFO fs.ObjectStoreVisitor: Stocator registered as cos for cos://hdfsVault.applogs/1 2021-03-26 17:00:29,885 INFO cos.COSAPIClient: Init : cos://hdfsVault.applogs/1 2021-03-26 17:00:31,073 WARN cos.COSAPIClient: file status 1 returned 404 -mkdir: Fatal internal error java.lang.NullPointerException at com.ibm.stocator.fs.common.ObjectStoreGlobber.glob(ObjectStoreGlobber.java:180) at com.ibm.stocator.fs.ObjectStoreFileSystem.globStatus(ObjectStoreFileSystem.java:544) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:353) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:250) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:233) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:104) at org.apache.hadoop.fs.shell.Command.run(Command.java:177) at org.apache.hadoop.fs.FsShell.run(FsShell.java:327) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.FsShell.main(FsShell.java:390)

[mrmadira]

Hi @nkurn

• What is the environment you are testing this from?
• What is the version of stocator?
• And what is the object store you are testing with? Is it IBM COS?

[nkurn]

Hi @mrmadira Yes I am using IBM COS

OS

NAME="Ubuntu" VERSION="16.04.3 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian

hadoop

Hadoop 3.2.2

stocator

stocator-1.1.4

did you see the core-site.xml I have added?

thanks.

[nkurn]

hdfs dfs -ls hdfs dfs -cp and other operations work fine.

[mrmadira]

WE don't have a repro for this. Please close until we get more details

[nkurn]

Hi, can you guys please let us know if we are missing any configuration ?

why is stocator checking for file status ": file status 1 returned 404" when creating a directory?

thanks a lot, Naga.

[nkurn]

Also, mkdir is working inside the spark-shell

scala> val statuses = fs.mkdirs(new Path(s"cos://TestVault1.mycos/d6")) statuses: Boolean = true

it is failing only when I use the shell command, do you think I am missing some hadoop/hdfs configuration?

[nkurn]

hdfs dfs -mkdir cos://TestVault1.mycos/d7 2021-06-02 00:07:58,829 INFO fs.ObjectStoreVisitor: Stocator registered as cos for cos://TestVault1.mycos/d7 2021-06-02 00:07:58,840 INFO cos.COSAPIClient: Init : cos://TestVault1.mycos/d7 2021-06-02 00:07:59,855 WARN cos.COSAPIClient: file status d7 returned 404 -mkdir: Fatal internal error java.lang.NullPointerException at com.ibm.stocator.fs.common.ObjectStoreGlobber.glob(ObjectStoreGlobber.java:180) at com.ibm.stocator.fs.ObjectStoreFileSystem.globStatus(ObjectStoreFileSystem.java:544) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:249) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:232) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103) at org.apache.hadoop.fs.shell.Command.run(Command.java:176) at org.apache.hadoop.fs.FsShell.run(FsShell.java:326) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)

[mrmadira]

Hi Naga -can you share your classpath that takes effect for HDFS?

[nkurn]

HADOOP_CLASSPATH=/src/hadoop-release/lib/:/src/hadoop-release/etc/hadoop/conf/:/src/hadoop-release/etc/hadoop/:/src/hadoop-release/share/hadoop/tools/lib/

[nkurn]

I am the root user when I run the shell FS commands, does it matter which user runs hdfs mkdir commands?

[nkurn]

on the spark-shell using fs.mkdirs call the accesser is getting the put request and is succeeding

{"server_name":"xxxx","remote_address":"xxxx","remote_user":"xxxx","timestamp_start":"1622645396705","timestamp_finish":"1622645397190","time_start":"02/Jun/2021:14:49:56 +0000","time_finish":"02/Jun/2021:14:49:57 +0000","request_method":"PUT","request_uri":"/TestVault1/tdir1/","protocol":"HTTP/1.1","status":200,"response_length":"0","request_length":"0","user_agent":"stocator 1.0.38-SNAPSHOT, aws-sdk-java/1.11.589 Linux/3.10.0-1160.21.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/25.242-b08 java/1.8.0_242 scala/2.11.12 vendor/Amazon.com_Inc. com.amazonaws.services.s3.transfer.TransferManager/1.11.589","request_latency":"485","request_id":"51dabdce-9374-472d-ac6f-86e70a91c1a8","request_type":"REST.PUT.OBJECT","interface_type":"s3","stat":{"client_wait":0.05,"storage_wait":0.491,"digest":0.0,"commit":460.31,"turn_around_time":0.0,"total_transfer":460.862,"pre_transfer":22.681,"post_transfer":0.321},"object_length":"0","version_name":"da7c586a-40c5-4f4d-ad95-a4b8db4946f7","version_transient":true,"delete_marker":false,"last_modified":"2021-06-02T14:49:56.729Z","last_changed":"2021-06-02T14:49:56.729Z","e_tag":"d41d8cd98f00b204e9800998ecf8427e","object_name":"tdir1/","vault_name":"TestVault1","is_secure":false,"principals":{"identity":"1d86d781-35fa-44a2-b327-322e3cb8723c@00000000-0000-0000-0000-000000000000","aws":"xxx"},"type":"http","format":1}

[nkurn]

when it fails on the FS shell there are 3 requests "REST.HEAD.OBJECT, REST.HEAD.OBJECT and REST.GET.VAULT

{"server_name":"xxx","remote_address":"xxx","remote_user":"xxx","timestamp_start":"1622645524767","timestamp_finish":"1622645524831","time_start":"02/Jun/2021:14:52:04 +0000","time_finish":"02/Jun/2021:14:52:04 +0000","request_method":"HEAD","request_uri":"/TestVault1/tdir2","protocol":"HTTP/1.1","status":404,"response_length":"277","user_agent":"stocator 1.0.38-SNAPSHOT, aws-sdk-java/1.11.199 Linux/3.10.0-1160.21.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/25.242-b08 java/1.8.0_242","request_latency":"64","request_id":"adf2cd5c-f73b-4841-be27-6ad5a24b0a40","request_type":"REST.HEAD.OBJECT","interface_type":"s3","object_name":"tdir2","error_code":"NoSuchKey","vault_name":"TestVault1","is_secure":false,"principals":{"identity":"1d86d781-35fa-44a2-b327-322e3cb8723c@00000000-0000-0000-0000-000000000000","aws":"xxx"},"type":"http","format":1} {"server_name":"xxx","remote_address":"xxx","remote_user":"xxx","timestamp_start":"1622645524848","timestamp_finish":"1622645524901","time_start":"02/Jun/2021:14:52:04 +0000","time_finish":"02/Jun/2021:14:52:04 +0000","request_method":"HEAD","request_uri":"/TestVault1/tdir2/","protocol":"HTTP/1.1","status":404,"response_length":"278","user_agent":"stocator 1.0.38-SNAPSHOT, aws-sdk-java/1.11.199 Linux/3.10.0-1160.21.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/25.242-b08 java/1.8.0_242","request_latency":"53","request_id":"71926599-adbd-4a47-8f74-303760727a88","request_type":"REST.HEAD.OBJECT","interface_type":"s3","object_name":"tdir2/","error_code":"NoSuchKey","vault_name":"TestVault1","is_secure":false,"principals":{"identity":"1d86d781-35fa-44a2-b327-322e3cb8723c@00000000-0000-0000-0000-000000000000","aws":"xxx"},"type":"http","format":1} {"server_name":"xxx","remote_address":"xxx","remote_user":"xxx","timestamp_start":"1622645524909","timestamp_finish":"1622645524966","time_start":"02/Jun/2021:14:52:04 +0000","time_finish":"02/Jun/2021:14:52:04 +0000","request_method":"GET","request_uri":"/TestVault1/?prefix=tdir2%2F&delimiter=%2F&max-keys=1&encoding-type=url","protocol":"HTTP/1.1","status":200,"response_length":"279","user_agent":"stocator 1.0.38-SNAPSHOT, aws-sdk-java/1.11.199 Linux/3.10.0-1160.21.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/25.242-b08 java/1.8.0_242","request_latency":"57","request_id":"a35b2aed-dceb-4c6d-9b99-0c759b8ade69","request_type":"REST.GET.VAULT","interface_type":"s3","vault_name":"TestVault1","is_secure":false,"principals":{"identity":"1d86d781-35fa-44a2-b327-322e3cb8723c@00000000-0000-0000-0000-000000000000","aws":"xxx"},"type":"http","format":1}

[nkurn]

user_agent seems to be a little different each time, can you please check the requests above and see if it makes sense

[mrmadira]

Seems you are on a very old version of stocator. Can you please try with latest version 1.1.3?

[mrmadira]

I think you will face a different behaviour on latest though; I can see it throws a FNF exception on latest. I'll check this further but until then I have a workaround for you: Create an empty directory say testdir on HDFS using hadoop fs -mkdir /testdir. Then do a hadoop fs -cp /testdir cos://mybucket.myservice/

[nkurn]

I tested the latest as well, it also fails.

thanks a lot for the workaround !! it is working. Please let me know once you have something for me to test. Thanks again for your help !!

[nkurn]

nested directories creation doesn't seem to work

hdfs dfs -mkdir /dir2/dir3 hdfs dfs -cp /dir2/dir3 cos://hdfsVault.mycos/

is there a way to make it work?

thanks, Naga.

[nkurn]

looks like I need to use 2 step process

hdfs dfs -mkdir /dir2 hdfs dfs -cp /dir2 cos://hdfsVault.mycos/ hdfs dfs -mkdir /dir3 hdfs dfs -cp /dir3 cos://hdfsVault.mycos/dir2/