remind101
You get disconnected from `emp run` when a lot of output is generated
The remote connection will die with an error message like this:
attaching to container (e413a2f40b7de61f84839a7b8fd0d52f3b25997f80afcf3c7c2eee6eea2af305): InvalidParameterException: Upload too large: 1049218 bytes exceeds limit of 1048576
status code: 400, request id: ed318bff-1116-11e8-beegoku :: ~/remind/r101-api ‹16b3fe1› »
Reproducible if running commands with a lot of output, or smth likewhile true ; do ; find / ; done. It's a bit jarring to have the session suddenly die. You've gotta log back in and get back into the interactive context you were in, and be really careful about the output you're generating. And if you mess up and generate too much output, you have to do it again.
Is there any way empire can be more resilient to running commands with a lot of output remotely?
To clarify the issue above, this can happen when Empire is configured to forward logs from attached containers to CloudWatch. There's a lot of ways that CloudWatch logs forwarding can fail (throttling, messages too big, stream too big, etc).
When we originally discussed implementing this, we agreed that failing closed would be best for security, so containers would die if logs couldn't be forwarded. I can see how that would get annoying pretty quickly in the case above.
We could potentially provide a boot flag to ensure that CloudWatch logs failures don't effect the session, which would be pretty easy to do.
