chef-provisioning-aws icon indicating copy to clipboard operation
chef-provisioning-aws copied to clipboard

Published 20 hours ago • verified publisher icon chef-boneyard

AWS VPC Provisioning Fails. Chef returns error that it is not able to find resource with resource ID

Open r4hulgupta opened this issue 8 years ago • 2 comments

Recently updated Chef-Client to 12.15.19 and started seeing this new error. I have been using my VPC creation recipes for long and now when I run it, Chef tries to check on AWS for a resource with it's ID but returns not found even when the resource exists on AWS. Please help! Chef Stacktrace Output:

Generated at 2016-11-08 23:14:40 +0000
AWS::EC2::Errors::InvalidNetworkAclID::NotFound: aws_network_acl[EG5988_SEC_21.55_ACL_01] (no AWS object id) (network_stack::network_stack line 170) had an error: AWS::EC2::Errors::InvalidNetworkAclID::NotFound: The networkAcl ID 'acl-d837ecbf' does not exist
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:375:in `return_or_raise'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:476:in `client_request'
(eval):3:in `create_tags'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/aws-sdk-v1-1.66.0/lib/aws/ec2/tag_collection.rb:64:in `create'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/aws-sdk-v1-1.66.0/lib/aws/ec2/resource_tag_collection.rb:88:in `[]='
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provider/aws_network_acl.rb:28:in `block (2 levels) in create_aws_object'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:293:in `block in retry_with_backoff'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/retryable-2.0.4/lib/retryable.rb:68:in `retryable'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:291:in `retry_with_backoff'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:298:in `retry_with_backoff'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provider/aws_network_acl.rb:27:in `block in create_aws_object'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:42:in `block in converge_by'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/mixin/why_run.rb:52:in `add_action'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/provider.rb:176:in `converge_by'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:41:in `converge_by'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provider/aws_network_acl.rb:19:in `create_aws_object'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:123:in `block in <class:AWSProvider>'
(eval):2:in `block in action_create'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/provider.rb:361:in `instance_eval'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/provider.rb:361:in `compile_and_converge_action'
(eval):2:in `action_create'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provider/aws_network_acl.rb:11:in `action_create'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/provider.rb:145:in `run_action'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource.rb:603:in `run_action'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:69:in `run_action'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:97:in `block (2 levels) in converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:97:in `each'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:97:in `block in converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:96:in `converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:669:in `block in converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:664:in `catch'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:664:in `converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:703:in `converge_and_save'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:283:in `run'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application.rb:284:in `run_with_graceful_exit_option'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application.rb:260:in `block in run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/local_mode.rb:44:in `with_server_connectivity'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application.rb:243:in `run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:464:in `sleep_then_run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:451:in `block in interval_run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:450:in `loop'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:450:in `interval_run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:434:in `run_application'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application.rb:60:in `run'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/bin/chef-client:26:in `<top (required)>'
C:/opscode/chefdk/bin/chef-client:68:in `load'
C:/opscode/chefdk/bin/chef-client:68:in `<main>'

>>>> Caused by AWS::EC2::Errors::InvalidNetworkAclID::NotFound: The networkAcl ID 'acl-d837ecbf' does not exist
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:375:in `return_or_raise'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:476:in `client_request'
(eval):3:in `create_tags'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/aws-sdk-v1-1.66.0/lib/aws/ec2/tag_collection.rb:64:in `create'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/aws-sdk-v1-1.66.0/lib/aws/ec2/resource_tag_collection.rb:88:in `[]='
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provider/aws_network_acl.rb:28:in `block (2 levels) in create_aws_object'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:293:in `block in retry_with_backoff'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/retryable-2.0.4/lib/retryable.rb:68:in `retryable'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:291:in `retry_with_backoff'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:298:in `retry_with_backoff'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provider/aws_network_acl.rb:27:in `block in create_aws_object'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:42:in `block in converge_by'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/mixin/why_run.rb:52:in `add_action'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/provider.rb:176:in `converge_by'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:41:in `converge_by'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provider/aws_network_acl.rb:19:in `create_aws_object'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:123:in `block in <class:AWSProvider>'
(eval):2:in `block in action_create'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/provider.rb:361:in `instance_eval'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/provider.rb:361:in `compile_and_converge_action'
(eval):2:in `action_create'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provider/aws_network_acl.rb:11:in `action_create'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/provider.rb:145:in `run_action'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource.rb:603:in `run_action'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:69:in `run_action'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:97:in `block (2 levels) in converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:97:in `each'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:97:in `block in converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:96:in `converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:669:in `block in converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:664:in `catch'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:664:in `converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:703:in `converge_and_save'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:283:in `run'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application.rb:284:in `run_with_graceful_exit_option'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application.rb:260:in `block in run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/local_mode.rb:44:in `with_server_connectivity'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application.rb:243:in `run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:464:in `sleep_then_run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:451:in `block in interval_run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:450:in `loop'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:450:in `interval_run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:434:in `run_application'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application.rb:60:in `run'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/bin/chef-client:26:in `<top (required)>'
C:/opscode/chefdk/bin/chef-client:68:in `load'
C:/opscode/chefdk/bin/chef-client:68:in `<main>'

r4hulgupta avatar Nov 08 '16 23:11 r4hulgupta

Works again after downgrading Chef client to 12.14.89.

UPDATE: Worked one time then again error. Gives errors for different things, like ACL not found with ID, Route Table not found with ID...

r4hulgupta avatar Nov 08 '16 23:11 r4hulgupta

So this issue doesn't seem to be due to any versions. It is intermittent and I can still see this. Sometimes Chef is not able to find the resource even if it is created, mostly seen when it tries to go and create Name tags to the resource. Any solution to this? Is there a way to put a little delay in retries?

r4hulgupta avatar Nov 10 '16 21:11 r4hulgupta