[Metricbeat] AWS metric start time and end time should align with the value of the metric's period

[kwinstonix]

Sometime metricbeat get empty metric result of EC2 instance. There is an issue in aws.GetStartTimeEndTime

• Version: 8.3.3
• Steps to Reproduce:

get aws ec2 instance metrics with the python script.

import boto3
import datetime

client = boto3.client("cloudwatch", region_name="ap-northeast-2")

period = 300
end = datetime.datetime.utcnow().replace(second=0)
start = end - datetime.timedelta(seconds=period * 1)

response = client.get_metric_data(
    MetricDataQueries=[
        {
            "Id": "id1",
            "MetricStat": {
                "Metric": {
                    "Namespace": "AWS/EC2",
                    "MetricName": "CPUUtilization",
                    "Dimensions": [
                        {"Name": "InstanceId", "Value": "your instance id"},
                    ],
                },
                "Period": period,
                "Stat": "Average",
            },
            "Label": "label1",
        },
    ],
    StartTime=start,
    EndTime=end,
)

print('now: ', datetime.datetime.utcnow())
print('end: ', end)
print(response["MetricDataResults"])

If end mod period is between 60s and 120s , there may be empty metric result.

now:  2022-08-09 13:51:12.032382
end:  2022-08-09 13:51:00.150606
[{'Id': 'id1', 'Label': 'label1', 'Timestamps': [], 'Values': [], 'StatusCode': 'Complete'}]

now:  2022-08-09 13:51:03.090935
end:  2022-08-09 13:51:00.091199
[{'Id': 'id1', 'Label': 'label1', 'Timestamps': [], 'Values': [], 'StatusCode': 'Complete'}]

• CloudWatch metricset use this function to get start and end timestamp.

https://github.com/elastic/beats/blob/62ddcc4022bbdfe0c01d5f4e9564adc0f2c89a59/x-pack/metricbeat/module/aws/utils.go#L22-L37

• According to the AWS API doc, we should make StartTime and EndTime align with period.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html#API_GetMetricData_RequestParameters

For better performance, specify StartTime and EndTime values that align with the value of the metric's Period and sync up with the beginning and end of an hour. For example, if the Period of a metric is 5 minutes, specifying 12:05 or 12:30 as StartTime can get a faster response from CloudWatch than setting 12:07 or 12:29 as the StartTime.

• How to fix this

-    endTime := time.Now()
+    endTime := time.Now().Truncate(period)

[kwinstonix]

@kaiyan-sheng , can you take a look at this issue?