NooBaa will not return HEAD for key=dir1/ if it was not created as dir1/ even if kes=dir1/f1 exists

[eshelmarc]

Environment info

• NooBaa Version: VERSION
• Platform: Kubernetes 1.14.1 | minikube 1.1.1 | OpenShift 4.1 | other: specify

Actual behavior

1.metadata for directory is ignored

Expected behavior

1.metadata should be set on directory object

Steps to reproduce

1.create a directory object directory-name/ that includes metadata

More information - Screenshots / Logs / Other output

[eshelmarc]

Also HEAD request for directory name that ends with / fails Oct-17 17:06:00.954 [Endpoint/930083] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Resource>/new3/dirtest1/</Resource><RequestId>kuvwd2ie-f126de-243</RequestId></Error> HEAD /new3/dirtest1/ {"host":"fin36p","user-agent":"aws-sdk-go/1.33.5 (go1.14.4; linux; amd64)","authorization":"AWS4-HMAC-SHA256 Credential=minioadmin/20211018/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=eecf8641fa3db7720e41889ed2546d56cb50c572dca33e1114fd9ce28d2585d3","x-amz-content-sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","x-amz-date":"20211018T000719Z"} Error: No such file or directory

[guymguym]

This is the code ref to where we "translate" keys that ends with a / to a file name key/.folder path

https://github.com/noobaa/noobaa-core/blob/88a72471f9e00bd656862f6297d2d387808a679a/src/sdk/namespace_fs.js#L1057

[eshelmarc]

I see that if first create dir/ it creates dir/.folder and get HEAD works for dir/  but if I create dir/f1 before creating dir/ get HEAD for dir/ fails, I assume because .folder was not created.

once the .folder is created I see that metadata is saved in .folder and is being returned with get HEAD

so the question is why .folder is not created if key is dir/f1 and for dual access why metadata is not save in dir/ instead of dir/.folder

[guymguym]

@eshelmarc Two separate things here:

1. Directories that we create in the FS as a side-effect of putting an object with a key that includes / are not considered "real" S3 objects and only exist in the FS, but should not be returned in list-objects and HEAD dir/ should also return NoSuchKey - which was added in PR #6410 and I still think is the right behavior from S3 perspective.

2. For dual access when S3 PUT dir/ works it sets both the meta-data and the data of the put request on the dir/.folder file instead of the directory, while it is possible to set the meta-data on the dir and only the data to a file. I agree that this might be a bit better because it's not very common that S3 PUT is used to send content for a dir/ key, although technically possible. Perhaps we can repurpose this issue to capture this desired change (update the issue title?)

[eshelmarc]

Just to summarize what we discussed. AFM/S3 will identify itself with var AddHostExecEnvUserAgentHander = request.NamedHandler{ Name: "AFM.1",

If NooBaa is using FS to store object as files. NooBaa will do the following for PUT of a directory "dir1/" It will not create .folder since data will not be provided and metadata will be set on the dir1/ directory.

If there is a a file that was put with PUT dir2/f1
The HEAD request "dir2/" should return success.