Project-MONAI
MIG Mongo Queue
Description MIG is currently using Mongo for persistence but it is also using it for resiliency. This caused an issue in SIT when we rebuilt the environment because as MIG started it started to read from an internal Mongo queue for data which no longer existed in minio. This caused errors in the logs but from a users perspective they would never know as MIG was still able to process incoming associations and added them to the bottom of its internal queue.
Suggestion is that moving the internal queue to rabbit is a better solution and provides more resiliency and observability to this issue making it easier to maintain.
Additional Information
@JoeBatt1989 have you tested the latest release https://github.com/Project-MONAI/monai-deploy-informatics-gateway/releases/tag/0.3.8?
In 0.3.8, at startup, MIG would clear any pending payloads that were created by the same instance by using the machine name.
And yes, the database is used for resiliency but it shall not affect the performance as an in-memory queue is used.
@mocsharp we were still seeing the issue on GSTT prod, I will gather some details and make sure we have the right deployment etc