Streaming device logs may loose log entries

[jellyfish-bot]

[fisehara] [fisehara] [fisehara] [fisehara] Device logs on dashboard and via balena cli are lacking entries when streamed with --tail option on CLI. All entries are existing when querying logs without --tail option on balena cli.

Most probably here the investigation need to start: https://github.com/balena-io/open-balena-api/blob/d32efff35e8a7689380b86ce0b6219c7d5dec65f/src/features/device-logs/lib/read.ts#L33

See: https://balena.zulipchat.com/#narrow/stream/346007-balena-io.2FbalenaCloud/topic/Random.20missing.20logs

[jellyfish-bot]

[fisehara] This has attached https://jel.ly.fish/4bfc9691-ae87-4101-b110-9d7ce0efa6dc

[jellyfish-bot]

[fisehara] Re-opening as the issue is occurring again.

[fisehara]

Redis PubSub Delivery semantics: https://redis.io/docs/interact/pubsub/#:~:text=Ctrl%2DC.-,Delivery%20semantics,-Redis%27%20Pub/Sub

The underlying redis PubSub structue only guarantees at-most-once so it's `expected`` to loose messages. redis-backend needs to implement redis streams: https://redis.io/docs/data-types/streams/