[cannot reproduce] Pagination cuts off logs

[ackava]

🔍 Check for existing issues

• [x] Completed

How is Dozzle deployed?

Standalone Deployment

📦 Dozzle version

v8.14.9

✅ Command used to run Dozzle

agent

🐛 Describe the bug / provide steps to reproduce it

Some logs are missing while seeing logs, I have attached screenshots

💻 Environment

Ubuntu 24 with latest Docker

📜 Dozzle logs are required for debugging purposes. You may need to enable debug mode. See https://dozzle.dev/guide/debugging.

I am seeing this one entry in dozzle-ui, but I dont' know if it is relevant.

{"level":"error","version":"v8.14.9","error":"unknown code: Unavailable with rpc error: code = Unavailable desc = error reading from server: EOF","container":"f1d111ede996","time":"2025-11-27T04:41:06Z","message":"unknown error while streaming logs"}

The container id is not same as the one where logs are missing, and there aren't any other error or warning items on dozzle-agent or dozzle-ui for every host.

📸 If applicable, add screenshots to help explain your bug

You can find actual logs after the same line in console

[ackava]

I think I found the issue, I keep on scrolling up and it looks like in slow speed, logs take time to download, so partial log is displayed if new log fetch request is sent before previous fetch completes.

After clearing cache, I scroll up slowly (letting previous request complete), then I don't see missing lines.

[amir20]

Hmm that's weird and possible a bug. Can you record a video with web tools open so I can see what's happening?

[ackava]

This is the log in the chrome debugger tools network pane,

This is when logs load.

This is when I scroll up.

This is when I scroll up again.

Do you see the gap in the time "from" and "to" from previous request? First request starts from 9.18 but second request's from doesn't end at 9.18.

[amir20]

I think I know what might be happening. The screenshot still make me wonder if it's a bug though.

My hunch is that you have logs all coming together so their timestamp is identical. Dozzle uses Docker API and it doesn't really have a clean API to fetch between two logs. I have to use timestamp instead. In your screenshot all logs are printed at the same time so when Dozzle tries to paginate it doesn't know where to cut off.

Now, the screenshots, I do wonder if there is a bug. 🤔 You said when you did it slow it wasn't happening though. So I wanted to see a video of that. Do you mean scrollup slow or waiting for each to load.

Did you scroll up and then very fast scroll up again? What's the best way to reproduce this for myself?

1. Does this happen to other logs too that are more spread? Or is it just this container?
2. What do you mean by slow?
3. Do you think if I print logs very close to each other would be a good way to reproduce?

[ackava]

1. Logs are missing from every container, in fact, many times we noticed that some logs are missing and we assumed that we have bug in our code and we went on whole week trying to figure out the bug, but when I compared to every other container, logs are missing.
2. It appears that logs are skipped when I scroll up, speed of scrolling up doesn't usually matter. But yes there are gaps.
3. Docker displays logs by UTC timestamp so, I think looking up history should have closest 1ms gap since the oldest timestamp received from the previous log fetch. I checked docker api and I feel this is the only solution.

[amir20]

Thanks. Any details you have helps. I am going to print 1000 lines of logs at the same time and see if it breaks. No one else has reported this so something feels unique.

[amir20]

Docker displays logs by UTC timestamp so, I think looking up history should have closest 1ms gap since the oldest timestamp received from the previous log fetch. I checked docker api and I feel this is the only solution.

I actually use nanoseconds.

So here is what I did:

1. I created a VM so that I can test with agent since that's what you are all doing.
2. I created a random container that prints 1000 lines at the same time.

docker logs ...
2025-11-29T17:20:35.362231521Z line 9981
2025-11-29T17:20:35.362232271Z line 9982
2025-11-29T17:20:35.362233021Z line 9983
2025-11-29T17:20:35.362233771Z line 9984
2025-11-29T17:20:35.362234521Z line 9985
2025-11-29T17:20:35.362235271Z line 9986
2025-11-29T17:20:35.362236021Z line 9987
2025-11-29T17:20:35.362236771Z line 9988
2025-11-29T17:20:35.362237521Z line 9989
2025-11-29T17:20:35.362238229Z line 9990
2025-11-29T17:20:35.362238979Z line 9991
2025-11-29T17:20:35.362239729Z line 9992
2025-11-29T17:20:35.362240479Z line 9993
2025-11-29T17:20:35.362241229Z line 9994
2025-11-29T17:20:35.362241937Z line 9995
2025-11-29T17:20:35.362242729Z line 9996
2025-11-29T17:20:35.362243437Z line 9997
2025-11-29T17:20:35.362244187Z line 9998
2025-11-29T17:20:35.362244937Z line 9999

3. I setup Dozzle Agent on the VM
4. I setup Dozzle UI on my laptop and connected to the agent
5. I loaded the test container and started scrolling up. Each log was loaded and when it got to the very top it loaded the previous logs exactly as expected.

See the video:

https://github.com/user-attachments/assets/140a72ee-de66-49d5-a90a-d0a3365aa865

I am not sure what else to try. 🤷

[ackava]

@amir20 my vms are remote, I am in India and all vms are in US east. Could you please enable bandwidth throttling in chrome debugger and see if you can reproduce the issue? And introduce some random delays in the logs to see if timing changes anything.

[amir20]

I'll give it a try. Although, I have logic to wait until pagination is done. If you want to help, then it would be great to figure out a way to easily reproduce. Dozzle is used by many people and this is a pretty sever bug in my opinion. So something is different. Looking at your screenshots, I can't make sense what's could be wrong.

[amir20]

https://github.com/user-attachments/assets/38a2ccd0-8c6d-4809-8df1-b630d491879c

Everything works on my side. Here is the video. I slowed it down to 4G. I also looked at the payload of the response and confirmed that it is returning up to a 100 logs per page.

[ackava]

I tried couple of times and I have noticed a pattern on when it is happening.

I have a backup schedule in a container that runs every 5 minutes. So there are too many logs in a couple of seconds, but for more than 4 minutes there are no logs. This container is running continuously, but the process inside it goes to sleep for 5 minutes. Logs of this container are skipped.

There is one container, which generates logs continuously every second, logs aren't skipped, all logs are displayed. This was the main database we usually don't notice, we only monitor the backup container and our application both of which has gaps in logs.

I was able to reproduce this on different machine with this https://github.com/social-mail/docker-postgres

This is a setup of postgres server with 5 minute incremental backup setup.

This is the compose file https://github.com/Social-Mail/docker-postgres/blob/main/deploy/primary/docker-compose..yml

I could setup a VM but I don't know how to securely send you credentials over github. This specific postgres installation requires public static IP with 80 port open to setup SSL.

But I guess it should be easy to reproduce this issue.

[amir20]

Nice! Now we are getting some where.

You have burst logging. This was something that I worked on another issue to improve performance issues. Dozzle tries to be efficient about it. It could be breaking something.

Here is what I need:

1. What's pattern of logs? Is it nothing for five minutes then 100s of log burst?
2. In your screenshot you had logs on multiple lines. Is that what they are?
3. No need for VM. But can you share logs? You can email me if it helps. I can't find try to create simulation by replaying the logs.
4. I'll see how easy it is to setup the compose. Getting the logs is easier.

Let me know. I can test when I wake up.

[ackava]

What's pattern of logs? Is it nothing for five minutes then 100s of log burst?

Yes

In your screenshot you had logs on multiple lines. Is that what they are?

Yes, Logger displays JSON with formatting so it is a single log string but contains new line characters in it.

No need for VM. But can you share logs? You can email me if it helps. I can't find try to create simulation by replaying the logs.

How do I do that? The best thing I can do is I can setup a temporary VM with the sample I have given.

If you cannot reproduce issue with the sample postgres compose I have provided, I will setup VM and will send credentials to your email.

[amir20]

I was hoping you can just export your logs using docker logs ... > test.log. I'll spend a little time this morning setting up you compose. I have to time box it though.

[amir20]

Your project expects AWS API keys which I don't have

InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.

So I won't have time to figure it out.

Just export the container that does the backup and send it to me exactly as Docker shows it.

[ackava]

@amir20 I will send you logs on weekend, I will send a dropbox download link for logs on your email address.

[Bayharbourbutcher1]

@amir20 I have the same issue with logs cutting off. The issue only occurs for me when i am connected to my home network remotely using wireguard in a VPS. As soon as i am back on the home network , the logs display correctly. I am on v8.14.9, let me know if i can help.

[amir20]

@ackava are you using VPN? When exporting make sure they have timestamp. docker logs -t should be easy.

@Bayharbourbutcher1 I am not sure if it is VPN related. Let's sees what @ackava says. If it is, I am not sure how I could help. There could be a lot happening when using a VPN. From dropped packets to timeouts or even changed payload. I am afraid that would be outside of my expertise to jump in and debug VPN issues. You can play around with different settings to see if you can find the culprit.

[amir20]

Going to close this. Open a discussion when you have logs ready.