s3-streamlogger icon indicating copy to clipboard operation
s3-streamlogger copied to clipboard

Suggestion to add a potential memory leak warning to the readme

Open moshetet opened this issue 5 years ago • 6 comments

S3StreamLogger will keep the logs in this.buffers until it needs to create a new file. So if setting large rotate_every and max_file_size you can bump into a nasty memory leak.

A warning should be added to the README to warn against this, especially since the following 2 things cause you to believe it is not a possibility unless you dig into the code of the package:

  1. The package says that it implements a stream (which usually means things don't stay in memory) after they are sent away.
  2. The options rotate_every and max_file_size just refer to the way the files are rotated in S3.

moshetet avatar May 30 '19 11:05 moshetet

In order to update an existing file in S3, this module re-uploads the entire file. Due to the way S3 works, it is not practical to do anything else. For this reason you should use a relatively small maximum file size.

autopulated avatar Jun 06 '19 14:06 autopulated

Once a file rotates or reaches it's max_file_size or rotates in S3, are the logs cleared from this.buffer?

Once a file rotates or reaches it's maximum do the logs on the application clear?

conor-deegan avatar Oct 27 '19 19:10 conor-deegan

Yes, buffers are cleared when a file is rotated due to reaching maximim age or size.

autopulated avatar Nov 01 '19 12:11 autopulated

Thank you!

conor-deegan avatar Nov 01 '19 15:11 conor-deegan

is there might be some potential logs losing with parallel executing?

vks-dbb avatar Jan 10 '24 15:01 vks-dbb

@vks-dbb Yes, different servers/processes should write log files to different files. The default name_format tries to ensure this.

autopulated avatar Jan 10 '24 16:01 autopulated