Support S3 Custom Protocol Tests

[aajtodd]

https://github.com/awslabs/smithy/blob/main/smithy-aws-protocol-tests/model/restXml/services/s3.smithy

We need not support all of them at once, but we should figure out how we are going to pull in these kind of customization tests (there are others for restJson1 and awsJson1_1).

Ideally they get generated with the specific service and are unit tests of that service.

@kggilmer perhaps additional things to ponder while considering service customizations

Acceptance criteria:

• service-based protocol tests are generated and pass

Protocol test docs: https://awslabs.github.io/smithy/1.0/spec/http-protocol-compliance-tests.html

[vinoddin]

By dev preview, we will generate all these tests, but may exclude a subset, and turn them back on as we add more S3 customizations.

[kggilmer]

A note for whoever takes this up, a quick change (to get started) to enable S3 protocol tests in our protocol-tests module:

diff --git a/codegen/protocol-tests/build.gradle.kts b/codegen/protocol-tests/build.gradle.kts
index a0a57b6..2aa66e3 100644
--- a/codegen/protocol-tests/build.gradle.kts
+++ b/codegen/protocol-tests/build.gradle.kts
@@ -92,7 +92,8 @@ val enabledProtocols = listOf(
     "aws-restjson",
     "aws-restxml",
     "aws-restxml-xmlns",
-    "aws-query"
+    "aws-query",
+    "s3"
 )
 
 enabledProtocols.forEach {
diff --git a/codegen/protocol-tests/smithy-build.json b/codegen/protocol-tests/smithy-build.json
index 2531bc2..96e8784 100644
--- a/codegen/protocol-tests/smithy-build.json
+++ b/codegen/protocol-tests/smithy-build.json
@@ -93,6 +93,29 @@
         }
       }
     },
+    "s3": {
+      "transforms": [
+        {
+          "name": "includeServices",
+          "args": {
+            "services": ["com.amazonaws.s3#AmazonS3"]
+          }
+        }
+      ],
+      "plugins": {
+        "kotlin-codegen": {
+          "service": "com.amazonaws.s3#AmazonS3",
+          "package" : {
+            "name": "aws.sdk.kotlin.servicetest.s3",
+            "version": "1.0"
+          },
+          "build":  {
+            "rootProject": true,
+            "optInAnnotations": ["software.aws.clientrt.util.InternalApi", "aws.sdk.kotlin.runtime.InternalSdkApi"]
+          }
+        }
+      }
+    },
     "aws-restxml-xmlns": {
       "transforms": [
         {

[kggilmer]

Implementation Estimation

• Implementing the codegen to integrate Smithy's AwsConfig structure into the Kotlin SDK's client configuration: 2 weeks
• Implementing virtual addressing styles: 1 weeks
• Defaulting to virtual host addressing: .5 weeks
• Implement dual-stack client configuration: 1 week
• Implement accelerate client configuration and addressing: 1 week
• Implement operation-level configuration: TBD

Total estimate is 5.5 weeks, without implementation of operation-level configuration.

[kggilmer]

Upstream dependency: https://github.com/awslabs/smithy-kotlin/issues/211

[kggilmer]

Reducing scope of issue to only handle codegen of service test, not any particular S3 functionality:

Implementing the codegen to integrate Smithy's AwsConfig structure into the Kotlin SDK's client configuration: 2 weeks

[aajtodd]

One of the issues with these tests (and any protocol tests for a service that has hand written components) is how to build the generated tests. The service customization protocol tests are written against a fake service model that has the same sdkId but only a small subset of operations built to test the bare minimum. This means that while it may trigger the same codegen logic, the resulting build is not the same.

The way we currently have it structured is to generate services to aws-sdk-kotlin/services. The root services/build.gradle.kts configures all of the subprojects. In particular it sets up the sourceSet(s) for generated-src as well as any hand written sources.

S3 for example has hand written code that is built alongside the generated sources (codegen integrations wire things up).

Protocol tests on the other hand do not share this build setup and do not know anything about services/s3/*. Our options are pretty limited here to either (1) heavily customize codegen to copy the sources and setup a resulting gradle build that does what the production build does. OR (2) make the custom sources a separate project that can be pulled in (or better yet inlined so as to not generate an actual dependency). This will move sources further away from where they are consumed and make it harder to reason about a particular service build at a glance.