[BUG] SigV4 signature by signer may be broken on retry

[sugymt]

What is the bug?

SigV4 signature by signer may be broken on retry. A retry request may fail with the following error log:

The request signature we calculated does not match the signature you
provided. Check your AWS Secret Access Key and signing method. Consult
the service documentation for details.

The Canonical String for this request should have been
...

How can one reproduce the bug?

Create a client that retries even when status 200.

// create signer in advance

client, _ := opensearch.NewClient(opensearch.Config{
	Address:       []string{"Opensearch domain URL"},
	RetryOnStatus: []int{200},
	RetryBackoff:  func(i int) time.Duration { return time.Duration(i) * 500 * time.Millisecond },
	MaxRetries:    5,
	Signer:        signer,
})

Create and run BulkIndexer on this client.

What is the expected behavior?

Retry requests successfully signed.

What is your host/environment?

go 1.19.3 opensearch-go v2.1.0 AWS ECS (Amazon Linux 2 / arm64 / ami-08011916b613f6c22)

Do you have any screenshots?

none

Do you have any additional context?

none

[dblock]

@sugymt Want to try and write a unit test for this? Are we dropping headers?

[sugymt]

@dblock I hope this issue will be fixed by some repository maintainer. I'm sorry but I can't solve it.

[dblock]

Maybe @VijayanB who wrote the original implementation can take a look?

[zethuman]

Hey there! @sugymt I can help to solve this issue, but I want more details to clarify what is going on.

• Do you have this trouble right after inserting data or after some time? (maybe token will be inactive by lifetime)
• Can you share with us how your generate payload and what is receiving, including headers? (I mean is this allowed by NDA)
• Which type of authentication do you use? IAM Auth or regular client to communicate with server?
• Maybe you want to send chunked request?

There is many cases what would be a reason of such trouble, to clarify and solve we need more details.

[tannerjones4075]

Is this issue still relevant? I would be interesting in helping.

[dblock]

@tannerjones4075 Likely yes!