beats icon indicating copy to clipboard operation
beats copied to clipboard

[Filebeat] Add health status reporting to AWS S3 input

Open taylor-swanson opened this issue 6 months ago • 2 comments

Update the AWS S3 input to report its health status, providing better visibility into the operational state for users.

The AWS S3 input should report its health status using the context.UpdateStatus() method. Status changes should only be reported when the status actually changes (don't repeatedly send HEALTHY if the status is already HEALTHY).

Status Reporting

The input should report the following states:

  • STARTING: When the input is initializing before attempting to connect to AWS.
  • CONFIGURING: When setting up AWS S3 client and validating configuration.
  • HEALTHY: When successfully connected to AWS S3 and monitoring buckets.
  • DEGRADED: When encountering non-fatal errors but can still operate. Examples:
    • Temporary network issues
    • Temporary permission issues that might resolve
  • FAILED: When encountering fatal errors preventing operation. Examples:
    • Authentication failures (invalid credentials)
    • Bucket not found
    • Insufficient permissions to access blobs
    • Persistent network connectivity issues

Include relevant context information with non-HEALTHY states:

  • For authentication issues: Include error message and status code (e.g., 401, 403)
  • For bucket not found: Include bucket name and error message
  • For connectivity issues: Include error message and endpoint being accessed

References

See the CEL input implementation for reference: https://github.com/elastic/beats/blob/6409d005a31e0147a40f0872794880b3df15a69c/x-pack/filebeat/input/cel/input.go#L111

taylor-swanson avatar Jun 04 '25 13:06 taylor-swanson

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

elasticmachine avatar Jun 16 '25 14:06 elasticmachine