cloudbeat
cloudbeat copied to clipboard
Convert formatted log messages to structured logging with ECS-compliant fields
This PR implements structured logging across AWS providers to replace formatted log messages with ECS-compliant structured fields, enabling better log aggregation, filtering, and monitoring while maintaining Fleet UI compatibility.
Changes Made
S3 Provider (internal/resources/providers/awslib/s3/provider.go)
- Converted 11 log statements from
log.Errorf()to structured logging usingWith()method - Added ECS-compliant fields:
aws.s3.bucket.name,cloud.region,error.message,aws.account.id - Implemented contextual loggers for bucket operations to reduce code duplication
- Updated both error and debug log statements
CloudTrail Logging Provider (internal/resources/providers/aws_cis/logging/provider.go)
- Converted 3 log statements to structured format
- Added fields:
aws.cloudtrail.trail.name,aws.s3.bucket.name,cloud.region,error.message - Implemented contextual logger for trail operations
IAM Providers (internal/resources/providers/awslib/iam/)
-
Root Account Provider: Converted 2 log statements with
aws.iam.user.name,error.messagefields - User Provider: Converted 4 log statements for MFA, password, and policy operations
Approach
Following the discussion in the issue comments, this implementation combines structured fields with existing formatted messages to:
- Maintain Fleet UI compatibility (original message format preserved)
- Enable enhanced monitoring and aggregation capabilities
- Use ECS-compliant field names for standardization
Example Output
Before:
{"message": "Could not get encryption for bucket MyBucket. Error: bla"}
After:
{
"message": "Could not get encryption for bucket MyBucket. Error: bla",
"aws.s3.bucket.name": "MyBucket",
"cloud.region": "us-east-1",
"error": {"message": "bla"}
}
Benefits
-
Enhanced filtering: Query specific buckets with
aws.s3.bucket.name:"bucket-name" -
Regional monitoring: Filter by region using
cloud.region:"us-east-1" -
Error categorization: Aggregate errors using structured
error.messagefield - Zero breaking changes: All existing tests pass, Fleet UI compatibility maintained
Validation
- ✅ All existing tests pass without modification
- ✅ Log output verified to contain both original message and structured fields
- ✅
go vetpasses with no issues - ✅ 20 log statements successfully converted across 4 provider files
Fixes #1888.
💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.