[Doc] On the subject of message handlers, threads, & coroutines

[Spydrouge]

• Scope: This section briefly documents some thoughts about signaling to the rest of the program that events have occurred, which will hopefully minimize errors and/or explain how we're minimizing errors.
• Origins: Our initialization functions were failing frequently because of problems with coroutines. I encountered many of them while trying to sense sent/received messages, and bumping into threads. In order to figure out how to resolve these issues sensibly, I jotted these notes.
• Purpose: to briefly lay out notes on the topics of networking and coroutines, in an attempt to reduce race conditions in which the system may mysteriously malfunction at mysterious times.
• Necessity: there are cleaner ways to do this, but because we do not really need this to scale (we are only using it for unit tests) and implementing a robust message handler system at this stage would be a lot of code to test/update/manage for pitifully small gain, I've just set down some rules of thumb for sensing that messages were sent. Some of these 'rules' simply originated from how Unity handles threads.
• Synopsis: Queue messages and send one per frame & record time sent. Do not presume that because a coroutine has started that it has also completed execution- especially lengthy initializers. Check for both.

Messages

Principle: Queue them It helps not send anything at the wrong time, especially when numerous parts of the program can generate message in coroutines, and the message handler is on a seperate thread where unity doesn't allow comparisons to occur.

Principle: Record that they happened The message handler is running on a seperate thread where unity heavily restricts what it can do. It can't even throw errors without causing catastrophe. Record the messages happen so the main thread can ping them.

Therefore: Wait a frame between processing or sending each message This prevents us from sensing two copies of the same message type and not having enough time to process that the first was sent. Ideally we would have a better messager, and queue the happenings, but our messages are so infrequent that tying them to the frame rate is harmless.

We give the main thread time to process each message set by the message handler thread. We also avoid creating much more code (this is a pretty limited function we're meeting here), or debugging its thread-safeness. However we need to document this 'principle' in case anyone wants to build something more comprehensive in its place.

Principle: Record times for time sensitive messages, not just booleans Helped us debug. Rather than using boolean variables to assess whether messages have been sent, save the times when messages are sent (and reduce them to longs for easy comparisons). This way, the system can more easily assess whether multiple messages of the same time have been sent, and ascertain the order that messages are sent in. Use highly precise time (ticks, saved as longs). This remains consistent with the use of DispatchTimes and ReceptTimes.

Therefore: Message times should not be 'cleared' or popped; they should always reflect the last dispatched/received message. Discovered upon comparing Update to FixedUpdate() and realizing some of our message receivers had no need to store times like these locally. A message should not be cleared by an external force; another external force might also want to see it. Neither should it be cleared by an update loop internal to the message handler, as a slower update system may want to check up on when the last time something updated was. The time each message last arrived should be retained until it is replaced by another message timestamp. Functions who care about whether a new message has been sent will record the old timestamp and use it for comparison purposes.

Note It may make more sense in the future to understand the gap between Update() and FixedUpdate(); By not clearing messages, we make it easier for our current, imperfect understanding of when exactly something updates to work just fine; but in the future FixedUpdate() and Update() and the frame-based nature of co-routines and messages have different rules behind when they proc; and we should make sure that both messages and message-pollers are updating at the same fixed rate so that they don't ever miss one another.

Therefore: Record like times for comparison The times recorded for messages are for comparative purposes (ie: to determine if two messages are the same, or to determine which message was sent first), so they must be recorded in a way that preserves message order, if not necessarily the exact times at which each message was sent.

Messages can be recorded as 'sent' at one of two times: when they are sent, or when they are queued. Currently, Opencog's messages are being recorded when they are queued. Because co-routines in Unity are not truly multi-threaded, we can be sure that the time recorded preserves the order the messages will be sent in, for comparison purposes against other messages (ie: was terrain perceived sent before or after mapinfo?)

They are timed at queuing because only because at present this is the easiest and most straightforward time at which they are identifiable by type. But in order to preserve the ability to compare different message times, all messages should now be recorded at queuing when it comes to dispatching messages.

Coroutines

Loose Principle: Understand the order Coroutines may start in, run in, and complete in. And that each set may be completely different. Small, swift, update functions which expect to build off of data laid down in other, large, initialization functions must check that those initialization functions have begun.

Therefore: Large order-dependent functions must track whether they've begun AND finished An example: the function PerceiveTerrain is a coroutine, but it must finish executing before PerceiveWorld is ever called, or else risk information from PerceiveWorld being overwritten by a pass of PerceiveTerrain. However, PerceiveTerrain must remain a coroutine; it will pause the game an unnecessary length of time otherwise.

Therefore, PerceiveTerrain must record when it has begun (so that multiple StartCoroutine(PerceiveTerrain()) calls are not made), and it must also record when it has ended (so that we know when we can start running PerceiveWorld)