s3 create bucket command results in ERR_STREAM_WRITE_AFTER_END error

[successkrisz]

Noticed that from something changed in how s3 commands are handled from 3.1.4 onwards.

When using:

• Serverless framework v3
• @aws-sdk/client-s3 nodejs library (using aws sdk v3)
• moto-server using the docker image

If I use [email protected] or below the following works:

import { PutObjectCommand, S3Client } from '@aws-sdk/client-s3'

const s3 = new S3Client({
  region:'eu-west-1',
  endpoint: 'http://localhost:5000',
  forcePathStyle: true,
})

s3.send(new CreateBucketCommand({ Bucket: 'bucketName' }))

If I update to any version of moto released beyond that the same thing fails without any code changes being made with the following error:

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at new NodeError (node:internal/errors:387:5)
    at ClientRequest.end (node:_http_outgoing:925:15)
    at writeBody (/Users/foo/sample-project/node_modules/@aws-sdk/node-http-handler/dist-cjs/write-request-body.js:22:21)
    at ClientRequest.<anonymous> (/Users/foo/sample-project/node_modules/@aws-sdk/node-http-handler/dist-cjs/write-request-body.js:9:13)
    at ClientRequest.emit (node:events:513:28)
    at ClientRequest.emit (node:domain:489:12)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (node:_http_client:622:11)
    at HTTPParser.parserOnHeadersComplete (node:_http_common:128:17)
    at Socket.socketOnData (node:_http_client:521:22)
    at Socket.emit (node:events:513:28) {
  code: 'ERR_STREAM_WRITE_AFTER_END',
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

I can't seem to find anything in the changelog or docs what would explain why this is happening, oddly enough if I replace the moto server with the latest localstack docker image it works 🤷‍♂️.

[bblommers]

Hi @successkrisz, thanks for raising this.

It looks like this is related to the dependencies packaged within the Docker-container. The same error occurs when running moto_server on the master-branch, with the latest version with flask==2.1.2 and werkzeug==2.1.2. However, if I install flask==2.0.3 and werkzeug==2.0.3 (the versions used in the Moto 3.1.3 docker container), everything works fine.

[bblommers]

Some notes on what I've found so far:

• The region matters - the error does not occur when the S3Client has the parameter region:'us-east-1'
• The error is due to werkzeug specifically - the flask-version does not matter
• Werkzeug 2.1.0 sends a HTTP/1.1 response - 2.0.3 sends a HTTP/1.0 response

Moto-specific findings:

• Moto sends a response-body, while AWS does not. Changing this behaviour does not make a difference - the error still occurs even when sending an empty body
• Moto sends some extra headers that AWS does not. Changing this does not make a difference either

Considering that the error still occurs after changing Moto's response to look exactly like AWS in terms of body/headers/HTTP-version, I'm a bit stumped on what else could be the cause.

[bblommers]

I had another look at this, and I believe it is a bug in how the either the AWS Node SDK or werkzeug handle 100-Continue requests.

Initially, the request only sends the headers. Only when a 100-continue event happens (i.e., the server sends a 100: Continue, the SDK sends the body. This happens here:

https://github.com/aws/aws-sdk-js-v3/blob/6b4bde6f338720abf28b931f8a4506613bd64d3f/packages/node-http-handler/src/write-request-body.ts#L9

However: This particular event is invoked twice, meaning that the body is send twice. That's what the error is referring to: ERR_STREAM_WRITE_AFTER_END talks about the request, not about Moto's response.

Locally I've verified this by patching the writeRequestBody-method in the linked file, to only send the request-body once:

httpRequest.on("continue", () => {
    if (!httpRequest.already_continued) {
        writeBody(httpRequest, request.body);
        httpRequest.already_continued = true
    }
});

With this change, the test works fine.

Logging in werkzeug (i.e., MotoServer) tells me that werkzeug only sends the 100-Continue once, so I don't quite understand why this event happens twice.

---

As a workaround: As of Moto 4.2.5, it is possible to run the MotoProxy instead. Using the following test:

import { CreateBucketCommand, S3Client } from '@aws-sdk/client-s3'
import { NodeHttpHandler } from '@aws-sdk/node-http-handler';
import ProxyAgent from 'proxy-agent';

const s3 = new S3Client({
  region:'us-east-2',
  forcePathStyle: true,

  requestHandler: new NodeHttpHandler({
        httpAgent: ProxyAgent('http://localhost:5005'),
        httpsAgent: ProxyAgent('http://localhost:5005')
    }),
})

s3.send(new CreateBucketCommand({ Bucket: 'asdfqwerasdfqwe3' }))

succeeds against the MotoProxy, without having to patch anything.

Note that this does involve setting up the required SSL certificates. The documentation has more information about this: http://docs.getmoto.org/en/latest/docs/proxy_mode.html

I configured Node by setting this environment variable:

export NODE_EXTRA_CA_CERTS=/location/to/ca.crt

I don't know enough about Node to know whether that is the best way to configure proxies/SSL validation - this just happened to work.