[v3.7.2] System.InvalidOperationException: Collection was modified in SimpleMessageStreamProducerExtension

[TheAngryByrd]

👋 Hey there,

We're running into the same issue as: https://github.com/dotnet/orleans/issues/7620. Unfortunately, the solution provided in this issue doesn't work for us because we make use of F#'s async computation expression and it may use a different scheduler. Rewriting this would prove difficult, fixing this in Orleans would be a better option for us.

---

The particular piece of code is here, where the enumeration is happening:

https://github.com/dotnet/orleans/blob/403fd6f0eeff4de48b373e28b7946b86a0a815fe/src/Orleans.Core/Streams/SimpleMessageStream/SimpleMessageStreamProducerExtension.cs#L155-L158

Since remoteConsumers.Values is being modified, it's failing.

We could add a .ToList() to copy the contents so that we no longer hit this issue. I don't think this is a problem because the call to RemoveRemoteSubscriber seems to be idempotent.

https://github.com/dotnet/orleans/blob/403fd6f0eeff4de48b373e28b7946b86a0a815fe/src/Orleans.Core/Streams/SimpleMessageStream/SimpleMessageStreamProducerExtension.cs#L183-L191