`put-bucket-policy` : An error message is not appropriate when an invalid JSON file is used.

[mosamu]

Environment info

• NooBaa Version: noobaa-core-5.16.0-20240229.el8.x86_64 + PR7848 patch

Actual behavior

Even if a JSON file is an invalid format, the aws output shows An error occurred (InvalidRequest) when calling the PutBucketPolicy operation: SOAP requests must be made over an HTTPS connection.

$ aws s3api --endpoint [https://localhost:443](https://localhost/) put-bucket-policy --bucket mosamu-b1 --policy [file://policy.json](file:///) --no-verify
/usr/lib/fence-agents/bundled/aws/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
 InsecureRequestWarning,

An error occurred (InvalidRequest) when calling the PutBucketPolicy operation: SOAP requests must be made over an HTTPS connection.

Expected behavior

The error message should be another one like Syntax error.

Steps to reproduce

N/A

More information - Screenshots / Logs / Other output

Here is a comment from @guymguym

In addition, the error message itself is just wrong and should be fixed - https://github.com/noobaa/noobaa-core/blob/29fc24a0a8ab0c2e38ea0d9af2db3fb070fb1aee/src/endpoint/s3/s3_errors.js#L207-L211 s3_errors.js

[romayalon]

@mosamu Can you please share the logs and the attached bucket policy? We throw MALFORMED_POLICY on several invalid policy checks, but we might missed one. Update - I was able to reproduce it locally when the content of the file was not json format. BTW, this is not a bug specific to NSFS NC (we get the bucket policy as the body of the request and fail before we even know it's NSFS flow)

[mosamu]

@romayalon Because of your update message, don't I have to share the log and policy file? Thanks,

[romayalon]

@mosamu Usually yes, it helps us understand the root cause of your specific scenario. I'll add here the info of my reproduction so you will be able to compare, if you see the exact same error in logs, you don't need to share it again, if you see other errors please provide the logs and the policy.

Here is the relevant info to the reproduction of the same s3api error code -

policy file -

% cat policy.json
this is not a json policy

s3api output -

AWS_ACCESS_KEY_ID=abc AWS_SECRET_ACCESS_KEY=123 aws s3api put-bucket-policy --bucket bucket1 --policy file://policy.json
An error occurred (InvalidRequest) when calling the PutBucketPolicy operation: SOAP requests must be made over an HTTPS connection.

NooBaa logs -

Mar-10 10:05:38.693 [nsfs/92351] [ERROR] CONSOLE:: parse_request_body: JSON parse problem SyntaxError: Unexpected token h in JSON at position 1
    at JSON.parse (<anonymous>)
    at parse_request_body (noobaa-core/src/util/http_utils.js:292:29)
    at Object.read_and_parse_body (noobaa-core/src/util/http_utils.js:241:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async handle_request (noobaa-core/src/endpoint/s3/s3_rest.js:149:5)
    at async Object.s3_rest [as handler] (noobaa-core/src/endpoint/s3/s3_rest.js:65:9)
Mar-10 10:05:38.694 [nsfs/92351] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidRequest</Code><Message>SOAP requests must be made over an HTTPS connection.</Message><Resource>/bucket1?policy</Resource><RequestId>ltl8ce6k-6kbbgy-2i0</RequestId></Error> PUT /bucket8318?policy {"host":"localhost:6443","accept-encoding":"identity","user-agent":"aws-cli/2.13.13 Python/3.11.5 Darwin/23.0.0 source/arm64 prompt/off command/s3api.put-bucket-policy",....} Error: SOAP requests must be made over an HTTPS connection.
    at parse_request_body (noobaa-core/src/util/http_utils.js:296:19)
    at Object.read_and_parse_body (noobaa-core/src/util/http_utils.js:241:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async handle_request (noobaa-core/src/endpoint/s3/s3_rest.js:149:5)
    at async Object.s3_rest [as handler] (noobaa-core/src/endpoint/s3/s3_rest.js:65:9)

[mosamu]

@romayalon Thank you for your advice, I intentionally recreated the error on my bench.

policy file - Although this is a JSON format, but it has an syntax error.

[mosamu@tcloud001 ~]$ cat invalid.json | jq
"Statement"
parse error: Expected string key before ':' at line 3, column 14

A bracket is missing.

[mosamu@tcloud001 ~]$ cat invalid.json 


  "Statement": [
      {
       "Sid": "AllowEveryoneReadOnlyAccess",
       "Effect": "Allow",
       "Principal": "*",
      "Action": [ "s3:GetObject", "s3:ListBucket" ],
      "Resource": ["arn:aws:s3:::mosamu-b1","arn:aws:s3:::mosamu-b1/*"]
        }
  ]
}

s3api output - The error must be same.

[mosamu@tcloud001 ~]$ aws s3api put-bucket-policy --endpoint http://localhost:80  --bucket mosamu-b1 --policy file://invalid.json                                                                

An error occurred (InvalidRequest) when calling the PutBucketPolicy operation: SOAP requests must be made over an HTTPS connection.

NooBaa logs -

2024-03-11T09:06:39.425027+09:00 tcloud001 node[63763]: [nsfs/63763] [ERROR] CONSOLE:: parse_request_body: JSON parse problem SyntaxError: Unexpected non-whitespace character after JSON at position 15    at JSON.parse (<anonymous>)    at parse_request_body (/usr/local/noobaa-core/src/util/http_utils.js:291:29)    at Object.read_and_parse_body (/usr/local/noobaa-core/src/util/http_utils.js:241:11)    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)    at async handle_request (/usr/local/noobaa-core/src/endpoint/s3/s3_rest.js:149:5)    at async Object.s3_rest [as handler] (/usr/local/noobaa-core/src/endpoint/s3/s3_rest.js:65:9)
2024-03-11T09:06:39.425705+09:00 tcloud001 node[63763]: [nsfs/63763] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidRequest</Code><Message>SOAP requests must be made over an HTTPS connection.</Message><Resource>/mosamu-b1?policy</Resource><RequestId>ltm6o9hf-82j83a-soc</RequestId></Error> PUT /mosamu-b1?policy {"host":"localhost","accept-encoding":"identity","content-md5":"mtsTNRqnwItc/ee4EUOUzA==","user-agent":"aws-cli/1.23.2 Python/3.6.8 Linux/4.18.0-477.27.1.el8_8.x86_64 botocore/1.23.46","x-amz-date":"20240311T000639Z","x-amz-content-sha256":"504237177988f988e9ddf681c9e6081b764262bd8b4bbd05905de7598129f816","authorization":"AWS4-HMAC-SHA256 Credential=CMAgCcoZOj1wYH3yV6L9/20240311/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=59e3419af640340bbf86c8fa2e500c4c0e4b7d630ff04c34ad4d504817d00fd7","content-length":"264"} Error: SOAP requests must be made over an HTTPS connection.    at parse_request_body (/usr/local/noobaa-core/src/util/http_utils.js:295:19)    at Object.read_and_parse_body (/usr/local/noobaa-core/src/util/http_utils.js:241:11)    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)    at async handle_request (/usr/local/noobaa-core/src/endpoint/s3/s3_rest.js:149:5)    at async Object.s3_rest [as handler] (/usr/local/noobaa-core/src/endpoint/s3/s3_rest.js:65:9)

Thanks,

[mosamu]

Hello, @romayalon I still see the same error at noobaa-core-5.15.2-20240425.el8.x86_64. Possibly noobaa-core-5.15.2-20240425.el8.x86_64 has not included this fix?

[user2@tcloud003 ~]$ myaws s3api put-bucket-policy --bucket user2-b1 --policy file:///home/policy.json

An error occurred (InvalidRequest) when calling the PutBucketPolicy operation: SOAP requests must be made over an HTTPS connection.

I used this invalid formrat JSON file.

[root@tcloud003 home]# cat policy.json | jq
"Statement"
parse error: Expected string key before ':' at line 1, column 14

[romayalon]

Hey @mosamu The fix was merged only yesterday, so indeed it's not in noobaa-core-5.15.2-20240425.el8.x86_64 try this one - noobaa-core-5.15.3-20240509-5.15.el8.x86_64.rpm I built it now on top of 5.15 branch.