Write to namespace buckets fail

[gbadanahatti]

Environment info

• NooBaa Version: NFO[0000] CLI version: 2.0.9
  INFO[0000] noobaa-image: noobaa/noobaa-core:5.2.11
  INFO[0000] operator-image: noobaa/noobaa-operator:2.0.9

and INFO[0000] CLI version: 2.0.10
INFO[0000] noobaa-image: noobaa/noobaa-core:5.2.13
INFO[0000] operator-image: noobaa/noobaa-operator:2.0.10

• Platform: Kubernetes 1.14.1

Actual behavior

PUTs to namespace buckets fail with a 500 error

Expected behavior

PUTs should succeed with a 200 return code

Steps to reproduce

More information - Screenshots / Logs / Other output

{"log":"\u001b[32mMar-11 18:33:17.328\u001b[35m [Endpoint/83] \u001b[31m[ERROR]\u001b[39m core.endpoint.s3.s3_rest:: S3 ERROR \u003c?xml version="1.0" encoding="UTF-8"?\u003e\u003cError\u003e\u003cCode\u003eInternalError\u003c/Code\u003e\u003cMessage\u003eWe encountered an internal error. Please try again.\u003c/Message\u003e\u003cResource\u003e/namespace-1/10m.file.\u003c/Resource\u003e\u003cRequestId\u003ek7nnvuh8-82uv4o-1165\u003c/RequestId\u003e\u003c/Error\u003e PUT /namespace-1/10m.file. {"host":"172.20.29.123:32653","accept-encoding":"identity","x-amz-content-sha256":"UNSIGNED-PAYLOAD","content-length":"10485760","user-agent":"aws-cli/1.14.28 Python/2.7.5 Linux/3.10.0-1062.12.1.el7.x86_64 botocore/1.8.35","expect":"100-continue","x-amz-date":"20200311T183317Z","content-md5":"7jIXdfV/jYbMQKOQXP22fQ==","authorization":"AWS4-HMAC-SHA256 Credential=bWNGoL1rie0JahVmrYRe/20200311/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=0e85c3af8f0f1032d60523e7f1eef23da32eacbf19ff20e6a2e398d0c66aa0cd"} Error: Non-file stream objects are not supported with SigV4\n","stream":"stdout","time":"2020-03-11T18:33:17.328251021Z"} {"log":" at Object.computeSha256 (/root/node_modules/noobaa-core/node_modules/aws-sdk/lib/util.js:740:23)\n","stream":"stdout","time":"2020-03-11T18:33:17.328273524Z"} {"log":" at Request.COMPUTE_SHA256 (/root/node_modules/noobaa-core/node_modules/aws-sdk/lib/event_listeners.js:143:18)\n","stream":"stdout","time":"2020-03-11T18:33:17.328277771Z"} {"log":" at Request.callListeners (/root/node_modules/noobaa-core/node_modules/aws-sdk/lib/sequential_executor.js:102:18)\n","stream":"stdout","time":"2020-03-11T18:33:17.328281213Z"} {"log":" at Request.emit (/root/node_modules/noobaa-core/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n","stream":"stdout","time":"2020-03-11T18:33:17.328284601Z"} {"log":" at Request.emit (/root/node_modules/noobaa-core/node_modules/aws-sdk/lib/request.js:683:14)\n","stream":"stdout","time":"2020-03-11T18:33:17.328288051Z"} {"log":" at Request.transition (/root/node_modules/noobaa-core/node_modules/aws-sdk/lib/request.js:22:10)\n","stream":"stdout","time":"2020-03-11T18:33:17.328297611Z"} {"log":" at AcceptorStateMachine.runTo (/root/node_modules/noobaa-core/node_modules/aws-sdk/lib/state_machine.js:14:12)\n","stream":"stdout","time":"2020-03-11T18:33:17.328301241Z"} {"log":" at /root/node_modules/noobaa-core/node_modules/aws-sdk/lib/state_machine.js:26:10\n","stream":"stdout","time":"2020-03-11T18:33:17.328304711Z"} {"log":" at Request.\u003canonymous\u003e (/root/node_modules/noobaa-core/node_modules/aws-sdk/lib/request.js:38:9)\n","stream":"stdout","time":"2020-03-11T18:33:17.328308089Z"} {"log":" at Request.\u003canonymous\u003e (/root/node_modules/noobaa-core/node_modules/aws-sdk/lib/request.js:685:12)\n","stream":"stdout","time":"2020-03-11T18:33:17.328311847Z"} {"log":" at Request.callListeners (/root/node_modules/noobaa-core/node_modules/aws-sdk/lib/sequential_executor.js:116:18)\n","stream":"stdout","time":"2020-03-11T18:33:17.328315974Z"} {"log":" at Request.emit (/root/node_modules/noobaa-core/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n","stream":"stdout","time":"2020-03-11T18:33:17.328319589Z"} {"log":" at Request.emit (/root/node_modules/noobaa-core/node_modules/aws-sdk/lib/request.js:683:14)\n","stream":"stdout","time":"2020-03-11T18:33:17.32832301Z"}

[nimrod-becker]

@gbadanahatti thanks for notifying us. Can you please attach the core logs and not just the snippet you shared? We would need it to verify what is the status of the bucket and resources.

Also, You can take a look in the UI and see if the bucket is healthy, as well as the resources you added to it

[gbadanahatti]

@nimrod-becker, attaching core logs. I have attached last 5000 lines of the logs. hopefully it can provide insights. 0_5000.log

Also, on the UI all resources are marked as healthy

[jeniawhite]

@gbadanahatti Hello, This is a known gap. You can take a look at https://github.com/noobaa/noobaa-core/issues/3642 to get more details. In AWS SDK you can take a look at https://github.com/aws/aws-sdk-js/issues/965 to get more details. Basically, the problem is with BodySigning. This is the code that decides the logic:

shouldDisableBodySigning: function shouldDisableBodySigning(request) {
    var signerClass = this.getSignerClass();
    if (this.config.s3DisableBodySigning === true && signerClass === AWS.Signers.V4
        && request.httpRequest.endpoint.protocol === 'https:') {
      return true;
    }
    return false;
  }

In file https://github.com/aws/aws-sdk-js/blob/cd2fe1e440bc33f77e16297fa4d0027a7b2dda72/lib/services/s3.js

This means that if you will use https endpoint it will work.

[nimrod-becker]

dup of 3642, closing