Pull working list messages at first ?

[fahchen]

The broadway is processing some messages, and then I restart it. When the system goes online later, the last working_list messages are left and never queued.

Should we provide a hook to queue the left working_list messages into list explicitly, or do it in producer implicitly? or any suggestions?

[amokan]

@fahchen Sorry for the delay! This is a good question and one I chose to not solve because, for me, it was an edge case that I couldn't solve with simple logic.

Depending on your message format in Redis, it is difficult to know which items are things that are currently being worked on or deserve to be moved back to the queue. This is true for anything using this 'durable queue' approach in Redis.

I find myself in situations where, sometimes, I have code written in another language working from the same queue. Because of that and if my messages in the list(s) do not contain any time information to determine age - It would be bad if I automatically moved the data from 'working' to back to my queue list.

I do use this library in production today and my system is pretty much running for weeks/months on end without being stopped. I monitor my working queue and have a script to move those items back from the working list to the queue list.

Having said that, I'm absolutely ok with someone implementing an optional hook and a PR to facilitate this as well. If I have cycles this week, I do plan to go in and update to Broadway 0.5 - so maybe wait on doing anything wild until you see that update.

[fahchen]

Thanks for your reply. My scenario is simpler than you now. Currently, I move working_list messages back to the list when my system is about going up manually. Maybe Broadway.Producer.prepare_for_start/2 in Broadway 0.5 is a nice hook to do this, we can expose this hook for users, now it's only available for producer.