OTEL Collector docker image logrotator.sh does not truncate file

[jezza323-kalibre]

I am running the currently tagged image "docker.hyperdx.io/hyperdx/hyperdx-otel-collector:2" in a basic virtualised RHEL docker environment for testing purposes. The logrotator.sh is running as expected, and appears to copy the log to agent.log.1 successfully, and delete it, but the original log file never truncates. This eventually blows out the file size on the disk and kills things

The command

> "$LOG_FILE"

Has no impact on the file size of the log file, even when running "> agent.log" manually from a docker exec into the running container. Other options suggested by googling such as

truncate -s 0 agent.log

have the same effect (eg none).

Is anyone else experiencing this? or is this an environment issue? I can create new files and truncate those successfully. I can also rm agent.log successfully. Not sure what else to look at here...

[brandon-pereira]

Hey @jezza323-kalibre 👋,

I tried to reproduce this locally but wasn’t able to — log rotation and deletion both worked as expected for me.

I also ran the > "$LOG_FILE" command manually in a sh environment, and it successfully truncated the file as expected (make sure you’re using sh rather than bash when testing this).

One thing to keep in mind: the logs are written very frequently, so if you cat the file, clear it, and then immediately cat it again, you might see new log lines that look identical to the old ones — but they’re actually newly written. To confirm, check the timestamps on the first line, e.g.:

head -1 /etc/otel/supervisor-data/agent.log

Example:

Could you double-check if that’s what’s happening in your environment?

[pulpdrew]

I can confirm the behavior @jezza323-kalibre is seeing in docker. The log file size keeps increasing, despite being rotated. Interestingly, if you cat the file, you only see the rows written since the last rotation attempt (so @brandon-pereira your head -1 command output is consistent with what I am seeing) but the file has a bunch of garbage data that shows up in a text editor, and ls -l shows the file size continuing to increase beyond the rotation size limit, despite truncation (with a brief period after the truncation where the file size is shown as 0).

I think we'll need to take a further look at what can be done to adjust or replace our logrotate script so that this works as expected. Thanks for reporting this @jezza323-kalibre

[jezza323-kalibre]

Thank you both for testing, and glad you were able to reproduce @pulpdrew . Makes me feel a bit less like I'm going insane 😬 This is not my area of expertise, but I will see if I can come up with a workable solution.

It does cause quite some disk thrashing copying the ever-growing file only to immediately delete the copy. Can probably skip the copy and delete if number of logs to retain is 1 and just truncate, provided I can get it to truncate at all

@brandon-pereira it is definitely not truncating at all, as the file grew to 115gb over a number of weeks (had it set to debug). I have manually cleaned up, updated log level and now running, but even so the current file 79mb as shown below. Yes I am using sh as bash doesnt seem to be available in the base image

and then manually ran

> agent.log

with the following result

[brandon-pereira]

@jezza323-kalibre @dhable is working on a fix for this in #1479, will ship in the next release. Please re-open this if there are still issues!

[linear[bot]]

from dan.hable:

@jezza323-kalibre we rolled this out to our internal environments and it did not behave as we expected. The change was reverted and we're going to continue to look into this.