Interested in finding out how you found the magic number 26 to add to message size

[mec07]

Hi,

Thanks for writing this package, it's really cool! I'm currently writing something similar for zerolog.

When you work out the total size of the batch to send to AWS you are taking into account the 1MB limit mentioned here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html In the code you write (https://github.com/kdar/logrus-cloudwatchlogs/blob/master/hook.go#L131):

messageSize := len(*p.Message) + 26

I'm just wondering where the 26 comes from and how you calculated it.

The way I would calculate it is by assuming that they're using json encoding and then counting the extra characters. When I do that, I arrive at a different number, 36. I do that by considering the json encoding to be:

{"message":"<len(*p.Message)>","timestamp":<int64, i.e. 8 bytes>},

The final comma will be present for all the messages except for the last one, so it's reasonable to count it. That looks like the total number of bytes per message would therefore be len(*p.Message) + 36 as there are 28 characters in the json (other than the message) and 8 bytes for the the timestamp.

I look forward to hearing back from you!

Marc

[kdar]

So that code change was introduced in https://github.com/kdar/logrus-cloudwatchlogs/pull/11 by @mattj103. I've been merging PRs based on good faith as I haven't used logrus/cloudwatch in years. I'll give them some time to respond, but if they don't, I'm happy to merge the change if you put up a PR.

[mattj103]

It's been a while but I believe it is from:

https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html

The maximum batch size is 1,048,576 bytes. This size is calculated as the sum of all event messages in UTF-8, plus 26 bytes for each log event.

The length I'm adding it to should be the message in UTF-8 so it doesn't care if it's JSON. Is it giving you some issue?

Thanks @kdar for pinging me, appreciate you keeping an eye on it still.

[mec07]

Ah, that's great, thanks! I really appreciate it. It also mentions the the 10k limit on messages. I was wondering if that was an AWS restriction or not.

The library I've written to do this for zerolog is here: https://github.com/mec07/cloudwatchwriter