Insane browser memory usage when looking at a container logs which generated hundreds of lines per second

[omasseau]

Describe the bug If I open the logs of a container which generates hundreds of logs per second (for example just after modifying the log level of an app to DEBUG, which could sometimes happen when needing to understand a problem) :

1. The browser is freezing
2. I can see the browser tab memory consumption increasing insanely (an increase of 11 gb in my case, in under a minute - 300mb per second on average) .

Before :

After :

The consumed memory is never released until I close the Dozzle tab in the browser (changing to another selected container - if possible - which does not generate logs is not enough)

To Reproduce Open the logs a a container which generates a huge amount of logs per second, by clicking on the container name Do nothing (don't scroll the logs up, just keep the latest logs displayed and see the panel content being refreshed with new lines of logs). Look at the browser memory consumption.

Expected behavior Consumed memory should be limited

Desktop (please complete the following information):

• OS: Windows 10
• Docker version :

 Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:33:07 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:10 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

• Browser & version : Firefox 124.0.1
• Version : 6.5.2

Logs screenshot :

[omasseau]

More infos: After analysing the containers generated logs, by downloading them, I can see that it generated ~1.500.000 characters per second (about 2MB of logs per second). So it seems maybe it is a a bit extreme, but still could happen occasionnaly.

[amir20]

Thanks for the info @omasseau! I think I can take a guess as to what is happening. This fix is complicated and probably not many people will run into this.

Currently, Dozzle keeps a buffer of 600 lines and discards older logs when buffer exceeds that size. That can be seen here. However, as you might have guessed, this buffer is based on number of lines and NOT number of characters. So 600 x 1.5m x2 Bytes is roughly 1.8GB. That's a lot.

I can't think of a good solution except to count every single character. I think your use cases is a rare one since I haven't heard anybody else having this issue.

[omasseau]

@amir20 Actually it is not 1.5m chars per line. 1.5 m is the total size generated in one second. Each line does not go above 5000 characters, but there can be many many lines generated in the same second.

[amir20]

I see, but that would still make sense. I don't really track number of characters. Just lines and hardcoded 600 lines as maximum buffer size.

[amir20]

@omasseau let me know if there is a better solution that you can think of...otherwise closing since there is nothing to do.