cloud-custodian icon indicating copy to clipboard operation
cloud-custodian copied to clipboard
verified publisher icon cloud-custodian

s3 filter check-public-block causes c7n to fail to scan remaining buckets when it encounters an access denied

Open cleo2525 opened this issue 4 years ago • 3 comments

Describe the bug When applying the s3 filter 'check-public-block' c7n fails when it encounters one bucket with "Access Denied" and does not continue to scan the remaining buckets

To Reproduce Create a bucket with a bucket policy that denies access to your c7n role Create a bucket with block public access disabled Run a c7n policy that contains the following filter

policies:
  - name: check-public-block
    resource: s3
    filters:
      - type: check-public-block

Expected behavior I expect c7n to log the Access Denied error for the restricted bucket, then continue to scan the remaining buckets in the account and flag the bucket with block public access disabled.

Background (please complete the following information):

  • OS: [e.g. OSX 10.15] macOS 10.15.7
  • Python Version: [e.g. python 3.8.1] Python 3.9.4
  • Custodian Version: [e.g. 0.8.46.1] 0.9.11
  • Tool Version: [if applicable]
  • Cloud Provider: [e.g. gcp, aws, azure] AWS
  • Policy: [please exclude any account/sensitive information]
policies:
  - name: check-public-block
    resource: s3
    filters:
      - type: check-public-block
  • Traceback: [if applicable, please exclude sensitive/account information]

2021-05-13 12:20:38,102: custodian.s3:WARNING Bucket:my-restricted-bucket unable to invoke method:get_bucket_location error:Access Denied 2021-05-13 12:20:38,433: custodian.s3:WARNING Bucket:my-restricted-bucket unable to invoke method:get_bucket_policy error:Access Denied 2021-05-13 12:20:38,484: custodian.s3:WARNING Bucket:my-restricted-bucket unable to invoke method:get_bucket_acl error:Access Denied 2021-05-13 12:20:38,535: custodian.s3:WARNING Bucket:my-restricted-bucket unable to invoke method:get_bucket_replication error:Access Denied 2021-05-13 12:20:38,581: custodian.s3:WARNING Bucket:my-restricted-bucket unable to invoke method:get_bucket_versioning error:Access Denied 2021-05-13 12:20:38,626: custodian.s3:WARNING Bucket:my-restricted-bucket unable to invoke method:get_bucket_website error:Access Denied 2021-05-13 12:20:38,670: custodian.s3:WARNING Bucket:my-restricted-bucket unable to invoke method:get_bucket_logging error:Access Denied 2021-05-13 12:20:38,714: custodian.s3:WARNING Bucket:my-restricted-bucket unable to invoke method:get_bucket_notification_configuration error:Access Denied 2021-05-13 12:20:38,855: custodian.s3:WARNING Bucket:my-restricted-bucket unable to invoke method:get_bucket_lifecycle_configuration error:Access Denied 2021-05-13 12:20:44,003: custodian.output:ERROR Error while executing policy Traceback (most recent call last): File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/c7n/policy.py", line 285, in run resources = self.policy.resource_manager.resources() File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/c7n/query.py", line 523, in resources resources = self.filter_resources(resources) File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/c7n/manager.py", line 111, in filter_resources resources = f.process(resources, event) File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/c7n/resources/s3.py", line 1415, in process if f.result(): File "/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 438, in result return self.__get_result() File "/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result raise self._exception File "/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/c7n/resources/s3.py", line 1424, in process_bucket config = s3.get_public_access_block( File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetPublicAccessBlock operation: Access Denied Traceback (most recent call last): File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/c7n/cli.py", line 352, in main command(config) File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/c7n/commands.py", line 125, in _load_policies return f(options, list(policies)) File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/c7n/commands.py", line 271, in run policy() File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/c7n/policy.py", line 1178, in call resources = PullMode(self).run() File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/c7n/policy.py", line 285, in run resources = self.policy.resource_manager.resources() File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/c7n/query.py", line 523, in resources resources = self.filter_resources(resources) File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/c7n/manager.py", line 111, in filter_resources resources = f.process(resources, event) File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/c7n/resources/s3.py", line 1415, in process if f.result(): File "/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 438, in result return self.__get_result() File "/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result raise self._exception File "/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/c7n/resources/s3.py", line 1424, in process_bucket config = s3.get_public_access_block( File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetPublicAccessBlock operation: Access Denied

/Users/user/github.com/cloud-custodian/cloud-custodian/custodian/lib/python3.9/site-packages/botocore/client.py(676)_make_api_call() -> raise error_class(parsed_response, operation_name)

  • custodian version --debug output

Please copy/paste the following info along with any bug reports:

Custodian: 0.9.11 Python: 3.9.4 (default, Apr 5 2021, 01:49:30) [Clang 12.0.0 (clang-1200.0.32.29)] Platform: posix.uname_result(sysname='Darwin', nodename='my-laptop', release='19.6.0', version='Darwin Kernel Version 19.6.0: Mon Apr 12 20:57:45 PDT 2021; root:xnu-6153.141.28.1~1/RELEASE_X86_64', machine='x86_64') Using venv: True Docker: False Installed:

argcomplete==1.12.2 attrs==20.3.0 boto3==1.17.33 botocore==1.20.33 importlib-metadata==3.7.3 jmespath==0.10.0 jsonpickle==1.3 jsonschema==3.2.0 pyrsistent==0.17.3 python-dateutil==2.8.1 pyyaml==5.4.1 s3transfer==0.3.6 setuptools==54.2.0 six==1.15.0 tabulate==0.8.9 typing-extensions==3.7.4.3 urllib3==1.26.4 zipp==3.4.1

Additional context Add any other context about the problem here.

cleo2525 avatar May 13 '21 17:05 cleo2525

One of the reasons could be - an ACL or policy on that specific bucket is not allowing your custodian role to scan it. You must check-in all other accounts and other S3 buckets- if you are getting the same ACCESS DENIED for all.

aakifshaikh avatar May 26 '21 15:05 aakifshaikh

Correct, we have bucket policy that denies c7n access to the bucket. The issue is, c7n does not gracefully fail on the 1 bucket and continue to scan the others. It crashes and fails to scan the other buckets in the AWS account, which are accessible.

cleo2525 avatar Jun 07 '21 21:06 cleo2525

Hey guys, Even I was getting similar issue which said Access denied api:GetPublicAccessBlock policy:CheckForPublicAclBlock-Off account:AWS_IT_SANDBOX region:us-east-1 c7n_org:INFO Policy resource counts Counter() I tried adding below IAM policy s3:GetBucketPublicAccessBlock But still I get same error Any suggestion ?

RamyaHosavalike avatar Jul 28 '22 14:07 RamyaHosavalike

I just encountered this today as well, and it appears there are similar issues with other S3 bucket filters like 'ownership' and 'bucket-encryption'. There is code that gracefully handles AccessDenied for some of the bucket augments, but not these filters.

I consider this to be a fairly major issue because we have developers that sometimes mistakenly get a little aggressive in their bucket Deny policies and, just one of those in an account completely breaks these policy executions.

mcripps9 avatar Oct 24 '22 21:10 mcripps9