Event Stream Support

[aajtodd]

Describe the feature

Tracking issue for full support of event streams.

• Spec
• API Design

Completed:

• [x] Codegen support for shape serialization/deserialization to the vnd.amazon.event-stream content type
• [x] Runtime support for framing of vnd.amazon.event-stream content type
• [x] Integration tests for serialization and deserialization of all message types (except unbound document members)
• [x] Lift all errors out of the event stream
• [x] Response streams (e.g. s3 select object) work
• [x] Track initial request signature in the execution context

Remaining body of work:

• [x] HTTP/2 full duplex stream support of underlying HTTP engine(s)
  • https://github.com/awslabs/smithy-kotlin/pull/694
  • https://github.com/awslabs/aws-sdk-kotlin/pull/687
• [x] Signing support for event stream request messages (details below) - #644 - https://github.com/awslabs/smithy-kotlin/pull/670
• [X] Provide explicit coroutine scope for operations to launch background work - https://github.com/awslabs/smithy-kotlin/pull/696 - https://github.com/awslabs/aws-sdk-kotlin/pull/689
• [ ] RPC support (initial request/response comes from the event stream rather than the HTTP request/response) - Deferred until later: https://github.com/awslabs/aws-sdk-kotlin/issues/693
• [x] Serialization/deserialization support for "unbound" document members (details below) - https://github.com/awslabs/aws-sdk-kotlin/pull/690
• [ ] Testing
  • [ ] Timeouts (add tests as necessary to verify our timeout configuration doesn't adversely affect streaming request/responses)
• [ ] Provide new examples for streaming operations
• [ ] Add service specific tests (integration, e2e, or replay) for all streaming operations

Signing

We are currently using the AwsSignatureType.HTTP_REQUEST_CHUNK

this results in string to sign of:

[INFO] [2022-03-09T01:35:43Z] [000070000cbd5000] [AuthSigning] - (id=0x7fae06a35dd0) Signing successfully built string-to-sign via algorithm SigV4, with contents                                                
AWS4-HMAC-SHA256-PAYLOAD                                                                                                                                                                                         
19731129T213309Z                                                                                                                                                                                                 
19731129/us-east-1/testservice/aws4_request                                                                                                                                                                      
be1f8c7d79ef8e1abc5254a2c70e4da3bfaf4f07328f527444e1fc6ea67273e2                                                                                                                                                 
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855                                                                                                                                                 
f2eba0f2e95967ee9fbc6db5e678d2fd599229c0d04b11e4fc8e0f2a02a806c6 

I believe what we actually want is AwsSignatureType.HTTP_REQUEST_EVENT (see here) which is not currently implemented.

The request body chunk signing hard codes the sha256 for an empty payload (e3b0c44298fc...). What we actually need is the sha256 of the :date header. See the transcribe streaming docs which document the signature process:

String stringToSign =
    "AWS4-HMAC-SHA256-PAYLOAD" +
    "\n" +
    DATE +
    "\n" +
    KEYPATH +
    "\n" +
    Hex(priorSignature) +
    "\n" +
    HexHash(nonSignatureHeaders) +
    "\n" +
    HexHash(payload);

Unbound document members

The @eventPayload event stream trait is supposed to be handled exactly like @httpPayload from the REST protocols, which means if there are members with @eventHeader and members with no trait (unbound), then the members with no trait should end up in the payload using the protocol format. For example:

@streaming
union FooEvents {
    event1: MessageWithUnboundPayloadMembers
}

structure MessageWithUnboundPayloadMembers {
    @eventHeader header1: Integer, // Ends up in event stream headers
    @eventHeader header2: Integer, // Ends up in event stream headers
    unbound1: String, // Ends up in payload
    unbound2: Integer // Ends up in payload
}

Currently no event streams are actually modeled this way.

Is your Feature Request related to a problem?

N/A

Proposed Solution

No response

Describe alternative solutions or features you've considered

No response

Acknowledge

• [X] I may be able to implement this feature request

AWS Kotlin SDK version used

N/A

Platform (JVM/JS/Native)

N/A

Operating System and version

N/A