Protocol error in kinesis putRecords in aws sdk > 3.81.0

[cesarfd]

Describe the bug

This is a reopening of #3602 as the bug is definitely not fixed as of v3.121.0

Expected Behavior

putRecords() should work correctly.

Current Behavior

Connection fails, error msg gives "Protocol error".

Reproduction Steps

Please refer to https://github.com/aws/aws-sdk-js-v3/issues/3602#issuecomment-1134973121

Possible Solution

No response

Additional Information/Context

No response

SDK version used

3.82.0 or later

Environment details (OS name and version, etc.)

Please refer to https://github.com/aws/aws-sdk-js-v3/issues/3602#issuecomment-1134973121

[gugu]

I confirm that it is not fixed in 3.128.0 Here is what I have in Sentry (not much)

It can be kinesis problem and not SDK, related: https://github.com/awslabs/amazon-kinesis-client/issues/711

[gugu]

@cesarfd let's try to debug. Do you use some other other AWS service, which does not support http2, for example DynamoDB? This error happens if someone uses http2 protocol for the endpoint, which does not support http2. And my guess aws sdk accidentally upgrades protocol version of the other service, and it fails

[cesarfd]

I can confirm it fails with and without using a custom VPC endpoint for Kinesis Streams (I tried it just in case it made a difference).

One thing that I don't understand is why is this not failing to everyone? AFAIK there should be nothing special that has to be configured in the client side or in the AWS console to use http/2 with any of the services, right?

[gugu]

Here is my workaround:

    const usKinesis = new KinesisClient({ region: "us-east-1", requestHandler: new NodeHttpHandler() });

[gugu]

My guess it is because we are the only people who use both http2-enabled AWS kinesis and http1-only AWS services together

[cesarfd]

Here is my workaround:

    const usKinesis = new KinesisClient({ region: "us-east-1", requestHandler: new NodeHttpHandler() });

I can confirm the workaround works. At least we can force it to use classic http 1.1

[RanVaknin]

@gugu Thanks for posting the workaround.

I added a p2 tag to it. Since it has a workaround other more pressing issues will take precedence, but this will be added to the team's backlog.

Thanks again, Ran

[300LiterPropofol]

NodeHttpHandler https://github.com/aws/aws-sdk-js-v3/issues/3780#issuecomment-1184381160 Hi @gugu, thank you for the post. I have one dumb question, where does the new NodeHttpHandler() come from? What require() do I need to import to use this handler or is it built-in in Nodejs? Thank you!

[dorrogeray]

@300LiterPropofol

import {NodeHttpHandler} from '@aws-sdk/node-http-handler'

[jomaora]

I have the same error on v3.405.0, with NodeJS v18.17.0

const document = // DATA TO SEND

const kinesis = new KinesisClient({
  endpoint: 'http://localhost:4567',
  credentials: {
    secretAccessKey: 'local',
    accessKeyId: 'local'
  },
  region: 'eu-west-1'
});

const command = new PutRecordCommand({
        StreamName: 'streamName',
        Data: Buffer.from(JSON.stringify(document)),
        PartitionKey: document._id
});

const result = await kinesis.send(command);

  Error (NghttpError) {
    $metadata: {
      attempts: 1,
      totalRetryDelay: 0,
    },
    code: 'ERR_HTTP2_ERROR',
    errno: -505,
    message: 'Protocol error',
  }

[a8t]

^ Me too, sdk 3.405.0 and node 20.6.0

[godu]

I have the same error on v3.405.0, with NodeJS v18.17.0

const document = // DATA TO SEND

const kinesis = new KinesisClient({
  endpoint: 'http://localhost:4567',
  credentials: {
    secretAccessKey: 'local',
    accessKeyId: 'local'
  },
  region: 'eu-west-1'
});

const command = new PutRecordCommand({
        StreamName: 'streamName',
        Data: Buffer.from(JSON.stringify(document)),
        PartitionKey: document._id
});

const result = await kinesis.send(command);

  Error (NghttpError) {
    $metadata: {
      attempts: 1,
      totalRetryDelay: 0,
    },
    code: 'ERR_HTTP2_ERROR',
    errno: -505,
    message: 'Protocol error',
  }

We found a workaround. It was because we used mhart/kinesalite to simulate Kinesis locally. Unfortunately, kinesalite doesn't support HTTP2 connect. We had to configure kinesis client to use HTTP1.1 instead.

import {NodeHttpHandler} from '@smithy/node-http-handler';
import {KinesisClient} from '@aws-sdk/client-kinesis';

const kinesis = new KinesisClient(
  KINESIS_ENDPOINT
    ? {
        endpoint: KINESIS_ENDPOINT,
        requestHandler: new NodeHttpHandler()
      }
    : {}
);

[a8t]

Sorry, i neglected to say that the same workaround worked for us:

import { KinesisClient } from '@aws-sdk/client-kinesis';
import { NodeHttpHandler } from '@smithy/node-http-handler';

const region = 'us-east-2';
const kinesisClient = new KinesisClient({
  region,
  requestHandler: new NodeHttpHandler(),
  maxAttempts: 5,
});

export default kinesisClient;

This workaround was mentioned earlier in the thread, but since then, AWS has moved some libraries under @smithy.

[cesarfd]

Still failing with @smithy/node-http-handler 2.1.10 and @aws-sdk/client-kinesis 3.465.0.

It only works when explicitly instantiating a NodeHttpHandler to the client.