Selective Debug Logging for Credentials Fetch Errors in Botocore

[psnilesh]

Describe the bug

The botocore.utils.py file currently has debug logs enabled for both the IMDS (Instance Metadata Service) fetcher and S3 operations. This is causing performance issues when trying to debug NoCredentialsFound errors, as the debug logs from S3 operations are also being generated, which can significantly impact application performance.

This is what I did to enable debug logs from credentials fetchers,

    boto3.set_stream_logger('botocore.utils', logging.DEBUG)
    boto3.set_stream_logger('botocore.credentials', logging.DEBUG)

Regression Issue

• [ ] Select this option if this issue appears to be a regression.

Expected Behavior

I would like to have a way to selectively enable debug logs only for the credentials fetch process, without enabling debug logs for other parts of the botocore library, such as S3 operations. This would allow me to effectively debug NoCredentialsFound errors without the performance impact of excessive debug logging.

Current Behavior

Too many logs

Reproduction Steps

Run below program and observe the amount of logs generated,

import boto3
import logging

boto3.set_stream_logger('botocore.utils', logging.DEBUG)
boto3.set_stream_logger('botocore.credentials', logging.DEBUG)

boto3.Session().client('s3').get_object(...) # 

Possible Solution

Introduce a new configuration option or a logging filter that would allow users to enable debug logging specifically for the credentials fetch process, without affecting the logging behavior of other parts of the botocore library.

Additional Information/Context

No response

SDK version used

Latest

Environment details (OS name and version, etc.)

Amazon Linux

[adev-code]

Hello @psnilesh, thanks for reaching out. Debug logs would include all and as much logs as possible to record every events that had happened. S3 like any other service, have their own APIs and specific actions that requires permission and in order to know where the fault has happened (e.g. what kind of Permission Action failed in the service), then it would include all logs from both SDK and the service side like S3.
As an alternative, you could filter by code as I have tried:

import boto3 
import logging 
import sys 

logging.basicConfig(level=logging.INFO, stream=sys.stdout)

class CredentialFilter(logging.Filter):
    def filter(self, record):
        # Allow logs that contain credential-related keywords
        keywords = ['credential', 'identity', 'role', 'token', 'auth', 'imds', 'metadata']
        message = record.getMessage().lower()
        return any(keyword in message for keyword in keywords)

utils_logger = logging.getLogger('botocore.utils')
creds_logger = logging.getLogger('botocore.credentials')

utils_logger.setLevel(logging.DEBUG)
creds_logger.setLevel(logging.DEBUG)

credential_filter = CredentialFilter()
utils_logger.addFilter(credential_filter)
creds_logger.addFilter(credential_filter)

s3 = boto3.client('s3')
try:
    response = s3.list_buckets()
except Exception as e:
    print(f"Error: {e}")

Please let me know if you have any questions. Thanks

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.