cfn-init - Unhandled exception during resume

[sagar-86]

Started seeing these errors right before cfn-signal command in cfn-init section of CloudFormation Template for a Windows EC2 Instance -

2022-07-15 05:41:24,318 [DEBUG] Running command 01-signalInitComplete
2022-07-15 05:41:24,333 [DEBUG] No test for command 01-signalInitComplete
2022-07-15 05:41:24,648 [ERROR] Unhandled exception during resume: [Errno 2] No such file or directory: 'C:\\Windows\\TEMP\\tmpq73uv2cm/subp-stderr-c6db8147-6a05-4f44-b523-83bdbe0ace3a.txt'
Traceback (most recent call last):
  File "cfn-init", line 73, in <module>
  File "cfnbootstrap\construction.pyc", line 187, in resume
  File "cfnbootstrap\construction.pyc", line 578, in run_config
  File "cfnbootstrap\construction.pyc", line 146, in run_commands
  File "cfnbootstrap\command_tool.pyc", line 116, in apply
  File "cfnbootstrap\util.pyc", line 587, in call
  File "cfnbootstrap\util.pyc", line 562, in call
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Windows\\TEMP\\tmpq73uv2cm/subp-stderr-c6db8147-6a05-4f44-b523-83bdbe0ace3a.txt'

Also seeing intermittently these errors -

2022-07-15 22:33:26,445 [INFO] Command 00-installCrowdStrikeAgent succeeded
2022-07-15 22:33:26,445 [DEBUG] Command 00-installCrowdStrikeAgent output:
2022-07-15 22:33:26,455 [INFO] Waiting 10 seconds for reboot
2022-07-15 22:33:36,471 [DEBUG] Running command 01-installNSGAgent
2022-07-15 22:33:36,471 [DEBUG] No test for command 01-installNSGAgent
2022-07-15 22:34:43,431 [ERROR] Unhandled exception during resume: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Windows\\TEMP\\tmp
0v5ql8c4\\subp-stderr-f102e1b6-609e-4bac-8bba-4f0221cbbb1f.txt'
Traceback (most recent call last):
  File "cfn-init", line 73, in <module>
  File "cfnbootstrap\construction.pyc", line 187, in resume
  File "cfnbootstrap\construction.pyc", line 578, in run_config
  File "cfnbootstrap\construction.pyc", line 146, in run_commands
  File "cfnbootstrap\command_tool.pyc", line 116, in apply
  File "cfnbootstrap\util.pyc", line 587, in call
  File "cfnbootstrap\util.pyc", line 568, in call
  File "shutil.pyc", line 740, in rmtree
  File "shutil.pyc", line 618, in _rmtree_unsafe
  File "shutil.pyc", line 616, in _rmtree_unsafe
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Windows\\TEMP\\tmp0v5ql8c4\\subp-stderr-f102e1b6-609e-4bac-8bba-4f02
21cbbb1f.txt'

Because of no cfn-signal SUCCESS replies, the stack is getting timed-out

[sagar-86]

I figured it out finally. I was using the public SSM parameter to use Latest AMI ID for Windows 2019, which got updated on 7/13.

Apparently the cfn-bootstrap module in latest AMI for Windows 2019 (Windows_Server-2019-English-Full-Base-2022.07.13) is hosed.

Used the latest minus 1 AMI for Windows 2019 (Windows_Server-2019-English-Full-Base-2022.06.15) and we are back in business. Another reason to not use public SSM parameters without testing.

Recommend AWS to fix since this is just a workaround and not a fix and many others are also observing the same!

[harveymonster]

@sagar-86, I ran into a similar problem with the Windows 2022 AMI and found another workaround that allowed me to use the July image. I noticed that the commands that were failing seemed to be ones that were single-lined but defined as multiline.

I had the following, which was working for months until I started to get the error you mentioned this month:

z-signal-success:
  command: !Sub |
    cfn-signal.exe --success true --stack ${AWS::StackName} --resource Instance --region ${AWS::Region}

I changed it to the following, and it started to work again:

z-signal-success:
  command: !Sub "cfn-signal.exe --success true --stack ${AWS::StackName} --resource Instance --region ${AWS::Region}"

I'm not sure how your commands were defined, so I can't tell if this would work for you or not.

[BEvgeniyS]

I had to work around this by reverting cfn-bootstrap to older version (taken from previously working AMI)

## This should be executed before running `cfn-init`
Remove-Item "C:\Program Files\Amazon\cfn-bootstrap\*"  -Force
Read-S3Object -BucketName "<YOUR BUCKET HERE>" -Key "cfn-bootstrap.zip" -File "C:\Windows\temp\cfn-bootstrap.zip"
Add-Type -AssemblyName System.IO.Compression.FileSystem
[System.IO.Compression.ZipFile]::ExtractToDirectory("C:\Windows\temp\cfn-bootstrap.zip", "C:\Program Files\Amazon\cfn-bootstrap\")

[villelahdenvuo]

I'm running into the same issue with cfn-init built on 14/6/2022.

Edit: For me the solution was to update to the latest cfn-bootstrap from https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-helper-scripts-reference.html#cfn-helper-scripts-reference-downloads

[mmaeng]

Thanks for your notice! This repo actually covers the open source cloudformation command line tool, which is separate from the cfn-bootstrap package that you are referring to.

This may be helpful https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/releasehistory-aws-cfn-bootstrap.html

If you have further questions please join us on AWS CloudFormation Discord