Check updating lambda's status is in progress or not

[kimsehwan96]

Issue #, if available: fixes: #968

Description of changes:

Add RetryPolicy for detecting failure of updating lambda occurred by 'LastUpdateStatus' and apply it.

        // RetryPolicy for preventing error occurred by LastUpdateStatus from updated lambda
        RetryPolicy<LambdaResponse> lambdaLastUpdateStatusRetryPolicy = new RetryPolicy<LambdaResponse>()
                .handleIf(throwable -> throwable.getMessage().startsWith("The operation cannot be performed at this time."))
                .withDelay(Duration.ofSeconds(5))
                .withMaxRetries(10)
                .onRetry(failure -> log.warn(String.join("", "Waiting for changing LastUpdateStatus [", functionConf.getFunctionName().getName(), "]")))
                .onRetriesExceeded(failure -> log.error("The LastUpdateStatus of updating lambda isn't change. Cannot continue."));

        // Make sure multiple threads don't do this at the same time
        synchronized (this) {
            return Failsafe.with(lambdaIamRoleRetryPolicy, lambdaLastUpdateStatusRetryPolicy).get(() ->
                    lambdaClient.updateFunctionConfiguration(updateFunctionConfigurationRequest));
        }
    }

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

I used ggp image built from this branch. It works well :)

[kimsehwan96]

@timmattison Hello, Can you review this PR?

This commit was my best approach to solve that issue (lambda update status).

I think may be there is other way to solve this problem

[kimsehwan96]

@timmattison When I first submit this PR, I didn't know that there is the lambda client waiter class in aws java sdk.

So I made checking logic for last update status with RetryPolicy

Now, I know that there is the waiter class for this use case. I fixed already submitted logic like below commit

(I tested this change in my local machine and it's working well)

ca4f5e3fc2cdf2caa3cbc2fb32e5f4e2b0eacbb4

Thank you.

[timmattison]

I tried working off of the commit you provided but I keep hitting this error:

[ERROR] AwsGreengrassProvisioner: The operation cannot be performed at this time. The resource arn:aws:lambda:us-east-1:xxx:function:xxx-HelloWorldPython3 is currently in the following state: 'Pending'. StateReasonCode: 'Creating' (Service: Lambda, Status Code: 409, Request ID: ffa1a66a-5a7a-4115-ae28-c9dba7bde55b, Extended Request ID: null)

Does it need the waiter and the retry policy?

[kimsehwan96]

@timmattison

Ah, it's weird. I will check there is any problem in this code.

Does it need the waiter and the retry policy?

No, It just need the waiter not the retry policy.

https://github.com/awslabs/aws-greengrass-provisioner/blob/ca4f5e3fc2cdf2caa3cbc2fb32e5f4e2b0eacbb4/src/main/java/com/awslabs/aws/greengrass/provisioner/implementations/helpers/BasicLambdaHelper.java#L267-L272

I added lambdaClient.waiter().waitUntilFunctionUpdated(getFunctionConfigurationRequest) and the getFunctionConfigurationRequest for that waitUntilFunctionUpdated method and I built a docker image with this code and tested it.

[timmattison]

Circling back on this I can't seem to get the new or the old version to work. Still going...

[timmattison]

I think I found the issue. Testing again now.

[kimsehwan96]

@timmattison Okay, If you need my local built image (from this branch), here is

https://hub.docker.com/layers/aws-greengrass-provisioner/sktpghks1/aws-greengrass-provisioner/latest/images/sha256-d591d95694a71514f2b1deff9557852a215d468a1dbfc73af3142b748b1d44e4?context=explore

[timmattison]

Can you take a look at #1014 and let me know if this works for you. I think the solution can be packaged up this way and it should work for both cases. What I was running into is that this code handles if a function goes into the pending state if it is being updated, but not if it is a new function.

[kimsehwan96]

I think your code is more suitable in most cases. (create new function, update existing function).

I will try this version tomorrow. Thank you.

[timmattison]

Thanks for your patience. Really hope this fixes it for you. Looks like a pattern I should be adopting in a few other places as well.

[kimsehwan96]

There is no problem with creating & updating lambda with this version in my case.

Thank you for your job. I use python a lot of time in my work not java, I learned good pattern with your PR's codes !

You can close this PR and related issue!