aws-cli icon indicating copy to clipboard operation
aws-cli copied to clipboard

make aws logs tail --follow ~2x faster

Open timotheecour4 opened this issue 2 years ago • 5 comments

Reduce sleep time which makes aws logs tail --follow ~2x faster (for lambdas that emit frequently enough, eg every second); before this PR, you often had a ~10s latency between the time a log is written to some lambda's stdout and the time it appears in terminal (5s buffer time for cloudwatch logs + _TIME_TO_SLEEP); after this PR this becomes ~6 seconds.

notes

untested: to further reduce time, maybe a lambda extension could be used, see Buffering configuration in https://docs.aws.amazon.com/lambda/latest/dg/runtimes-logs-api.html

timeoutMs – The maximum time (in milliseconds) to buffer a batch. Default: 1,000. Minimum: 25. Maximum: 30,000.

timotheecour4 avatar Jun 10 '22 00:06 timotheecour4

Hi @timotheecour4,

Thanks! This was originally 1 second and bumped to 5:

https://github.com/aws/aws-cli/commit/7d0981dc65959a04421d6fe5efbb9dd249aef198

We'd need to investigate why that change was made and if it's still relevant here before we can change it back.

kdaily avatar Jun 15 '22 23:06 kdaily

/cc @kyleknap and @jamesls since you were in that above commit

timotheecour4 avatar Jun 22 '22 05:06 timotheecour4

ping on this

timotheecour4 avatar Jul 15 '22 22:07 timotheecour4

ping, is there anything else needed on my end?

timotheecour4 avatar Aug 18 '22 20:08 timotheecour4

@timotheecour4 I think this could be a parameter or environment variable, because this can result in API DDOS

orlovmyk avatar Aug 30 '22 17:08 orlovmyk