open-balena-api
open-balena-api copied to clipboard
Streaming device logs may loose log entries
[fisehara] [fisehara] [fisehara] [fisehara] Device logs on dashboard and via balena cli are lacking entries when streamed with --tail
option on CLI.
All entries are existing when querying logs without --tail
option on balena cli.
Most probably here the investigation need to start: https://github.com/balena-io/open-balena-api/blob/d32efff35e8a7689380b86ce0b6219c7d5dec65f/src/features/device-logs/lib/read.ts#L33
See: https://balena.zulipchat.com/#narrow/stream/346007-balena-io.2FbalenaCloud/topic/Random.20missing.20logs
[fisehara] This has attached https://jel.ly.fish/4bfc9691-ae87-4101-b110-9d7ce0efa6dc
[fisehara] Re-opening as the issue is occurring again.
Redis PubSub Delivery semantics: https://redis.io/docs/interact/pubsub/#:~:text=Ctrl%2DC.-,Delivery%20semantics,-Redis%27%20Pub/Sub
The underlying redis PubSub structue only guarantees at-most-once
so it's `expected`` to loose messages.
redis-backend needs to implement redis streams: https://redis.io/docs/data-types/streams/