sematic icon indicating copy to clipboard operation
sematic copied to clipboard

Add "chunking" and logrotation to log streaming

Open augray opened this issue 2 years ago • 1 comments

Currently we upload the entirety of a sematic function's logs at once. We buffer them on disk. For long-running jobs, this may not be viable; the container may eat up too much of its allowed storage. We need to make it so we truncate logs after each upload so that the log file on disk doesn't get too large. But we'll need to be careful we don't lose any log lines when we do this.

augray avatar Sep 21 '22 23:09 augray

it turns out that there is no safe way to truncate without risking lost logs while streaming from a shared file. However, if we do this over a pipe, the ingestor will be directly intercepting the stream from the producer and can safely chunk things based off of line breaks as the stream is written to.

augray avatar Oct 06 '22 15:10 augray