cloudbeat icon indicating copy to clipboard operation
cloudbeat copied to clipboard

Convert formatted log messages to structured logging with ECS-compliant fields

Open Copilot opened this issue 7 months ago • 3 comments

This PR implements structured logging across AWS providers to replace formatted log messages with ECS-compliant structured fields, enabling better log aggregation, filtering, and monitoring while maintaining Fleet UI compatibility.

Changes Made

S3 Provider (internal/resources/providers/awslib/s3/provider.go)

  • Converted 11 log statements from log.Errorf() to structured logging using With() method
  • Added ECS-compliant fields: aws.s3.bucket.name, cloud.region, error.message, aws.account.id
  • Implemented contextual loggers for bucket operations to reduce code duplication
  • Updated both error and debug log statements

CloudTrail Logging Provider (internal/resources/providers/aws_cis/logging/provider.go)

  • Converted 3 log statements to structured format
  • Added fields: aws.cloudtrail.trail.name, aws.s3.bucket.name, cloud.region, error.message
  • Implemented contextual logger for trail operations

IAM Providers (internal/resources/providers/awslib/iam/)

  • Root Account Provider: Converted 2 log statements with aws.iam.user.name, error.message fields
  • User Provider: Converted 4 log statements for MFA, password, and policy operations

Approach

Following the discussion in the issue comments, this implementation combines structured fields with existing formatted messages to:

  • Maintain Fleet UI compatibility (original message format preserved)
  • Enable enhanced monitoring and aggregation capabilities
  • Use ECS-compliant field names for standardization

Example Output

Before:

{"message": "Could not get encryption for bucket MyBucket. Error: bla"}

After:

{
  "message": "Could not get encryption for bucket MyBucket. Error: bla",
  "aws.s3.bucket.name": "MyBucket", 
  "cloud.region": "us-east-1",
  "error": {"message": "bla"}
}

Benefits

  • Enhanced filtering: Query specific buckets with aws.s3.bucket.name:"bucket-name"
  • Regional monitoring: Filter by region using cloud.region:"us-east-1"
  • Error categorization: Aggregate errors using structured error.message field
  • Zero breaking changes: All existing tests pass, Fleet UI compatibility maintained

Validation

  • ✅ All existing tests pass without modification
  • ✅ Log output verified to contain both original message and structured fields
  • go vet passes with no issues
  • ✅ 20 log statements successfully converted across 4 provider files

Fixes #1888.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot avatar Jul 17 '25 15:07 Copilot