Conflict contract tests: contract_update_non_existent_resource and contract_update_create_only_property

[liyujiel]

Hey,

Our team is failing with contract test contract_update_non_existent_resource due it use contract_update_create_only_property test json file as previous resource model. This will make a failure loop that:

If test update the resource with same resource name, contract_update_create_only_property will failed due there is no createonly property change.

if test update the resource with different resource name, contract_update_non_existent_resource will failed due CfnNotUpdatableException instead of NotFound.

Let me know if you wanna me clarify something, thanks! Alex

[anshikg]

hey Alex, can send out the stack trace for contract test for these test failures. I have not seen this issue before and would like to see what the stack trace looks like.

[liyujiel]

Hey sure.

Below it's the lambda local log which I test with contract_update_non_existent_resource

Starting the Local Lambda Service. You can now invoke your Lambda Functions defined in your template through the endpoint.
2020-09-02 13:28:04  * Running on http://127.0.0.1:3001/ (Press CTRL+C to quit)
Invoking software.amazon.events.archive.HandlerWrapper::handleRequest (java8)
Decompressing aws-resource/target/aws-resource-handler-1.0-SNAPSHOT.jar

Fetching lambci/lambda:java8 Docker container image......
Mounting /private/var/folders/y1/nnb9xqkd5xgb6wnv2q08b_sjr6pnd7/T/tmpghsh4sx3 as /var/task:ro,delegated inside runtime container
START RequestId: 6e894376-1b8f-1cfe-72c4-0580869acae2 Version: $LATEST
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[UPDATE] invoking handler...
Update resource module: ResourceModel(archiveName=testArchiveName,...)
Previous Resource Model ResourceModel(archiveName=testArchiveName1, ...).
Resource of type 'AWS::Archive' with identifier 'ArchiveName' is not updatable with parameters provided. in a UPDATE action on a null: software.amazon.cloudformation.exceptions.CfnNotUpdatableException: Resource of type 'AWS::Events::Resource' with identifier 'ArchiveName' is not updatable with parameters provided.
software.amazon.cloudformation.exceptions.CfnNotUpdatableException: Resource of type 'AWS::Resource' with identifier 'ArchiveName' is not updatable with parameters provided.
	at software.amazon.events.archive.UpdateHandler.verifyNonUpdatableFields(UpdateHandler.java:82)
	at software.amazon.events.archive.UpdateHandler.handleRequest(UpdateHandler.java:42)
	at software.amazon.events.archive.BaseHandlerStd.handleRequest(BaseHandlerStd.java:19)
	at software.amazon.events.archive.BaseHandlerStd.handleRequest(BaseHandlerStd.java:12)
	at software.amazon.events.archive.HandlerWrapper.invokeHandler(HandlerWrapper.java:78)
	at software.amazon.events.archive.HandlerWrapper.invokeHandler(HandlerWrapper.java:39)
	at software.amazon.cloudformation.LambdaWrapper.wrapInvocationAndHandleErrors(LambdaWrapper.java:350)
	at software.amazon.cloudformation.LambdaWrapper.processInvocation(LambdaWrapper.java:320)
	at software.amazon.cloudformation.LambdaWrapper.handleRequest(LambdaWrapper.java:207)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at lambdainternal.EventHandlerLoader$StreamMethodRequestHandler.handleRequest(EventHandlerLoader.java:356)
	at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:902)
	at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:341)
	at lambdainternal.AWSLambda.<clinit>(AWSLambda.java:63)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:150)

You can see it actually use input with previous resource model which should not.

[liyujiel]

In our case, the Archive Name is create only property.

[PatMyron]

Also failing contract_update_non_existent_resource due to handlers being called with invalid models and expecting CfnNotFoundException as discussed in https://github.com/aws-cloudformation/cloudformation-cli/issues/527. I don't think it makes sense to use inputs/inputs_1_invalid.json as the input to contract_update_non_existent_resource and expecting CfnNotFoundException because services should be doing model validation before actually checking resource existence

[anshikg]

Also failing contract_update_non_existent_resource due to handlers being called with invalid models and expecting CfnNotFoundException as discussed in #527. I don't think it makes sense to use inputs/inputs_1_invalid.json as the input to contract_update_non_existent_resource and expecting CfnNotFoundException because services should be doing model validation before actually checking resource existence

Hey Pat, I agree with what you are saying. But, there was a discussion about removing invalid tests in the past and we decided otherwise. Here is a link to the PR where this was discussed: https://github.com/aws-cloudformation/cloudformation-cli/pull/530

[liyujiel]

Hey Anshika,

Thanks for reply. Could you confirm is contract test issue and won't block our resource launch if the contract test contract_update_non_existent_resource failed? Still, by the test name and description I don't think the input request should contains previous model.

Let me know what you think.