SeqCst as a default atomic ordering considered harmful

[HadrienG2]

WARNING: Past experience on Rust bugtrackers has taught me that this topic may generate heated discussion (among the few experts that know/care about it). Please avoid kneejerk responses and hear what I have to say before replying.

As currently constructed, the "Atomics" section of the Nomicon basically suggests users of atomics to go for SeqCst ordering as the safest option, and go for weaker orderings if and only if they can prove that 1/they are good enough for the job at hand and 2/they provide a significant performance benefit.

I already read this rationale before, and see where it is coming from. But I would nonetheless like to join @jeehoonkang's research team in disagreeing with inconsiderate use of SeqCst atomic memory ordering, and recommending instead that this atomic ordering be treated as a specialist tool for very specific use cases that should never be used in everyday code. Here is why.

---

One very important point about atomics-based synchronization, which is perhaps insufficiently stressed in teaching material, is that sometimes they are simply not the right tool for the job at hand.

Any atomics-based synchronization transaction (including one carried out with SeqCst ordering) is only complete when thread A has atomically read a value from memory that was atomically written to by thread B. As long as thread B's writes are "in flight", any non-atomic read of thread A into memory that is being non-atomically written by thread B is a data race. Conversely, if thread B continues writing non-atomically to shared data after sending the synchronization signal to thread A, it is also a data race.

If this is a problem, then atomics are not the right tool for the job, and blocking execution synchronization (like Mutex and CondVar) must be employed.

---

Many atomic orderings easily "fall out" from this load/store-based synchronization perspective:

• Relaxed ordering synchronize accesses to the atomic variable alone, excluding synchronization of any other data manipulated by the current thread (including other atomics). Any reasoning about program execution ordering which is based on shared memory accesses to other data than the atomic variable at hand is incorrect without further synchronization.
• Acquire/Release ordering provide message-passing-style synchronization. If a thread doing an Acquire read sees a value that was emitted as part of a Release write, then it also sees any other write to memory that was carried out by the "sender" thread before the Release write. It may also see more writes, of course, which is why it's important to stop writing to the "sent message data" after a Release write.
• Consume could also be explained in terms of message passing, if its hardware-inspired semantics were not so badly broken at the compiler level that Rust declined to expose it.

SeqCst ordering, however, is more complicated:

• The key guarantee that it provides is that all threads will agree on a single total order for all SeqCst writes as long as they use SeqCst reads to read from the same variables. This differs from Relaxed's basic ordering guarantees only in the fact that atomic accesses to different variables are ordered with respect to each other.
• SeqCst loads behave like Acquire loads and SeqCst stores behave like Release stores. However, contrary to popular belief SeqCst is not a magical trick that will give Release semantics to loads or Acquire semantics to stores. Doing so is fundamentally impossible at the hardware level. Therefore, using the same SeqCst terminology for loads and stores is an ergonomics footgun, as the synchronization guarantees are not the same. From this point of view, it would have been much better to provide separate SeqCstAcq, SeqCstRel and SeqCstAcqRel orderings, but alas it is too late to fix the C++11 memory model.

The only cases in which SeqCst's guarantees matter is when the correctness of the synchronization protocol depends on all threads agreeing on the precise interleavings of atomic accesses during execution. Few synchronization protocols truly need this guarantee, and in fact I treat code which relies on it with high suspicion because proving it correct requires checking all possible execution interleavings. As for N instructions there are N! interleavings, this quickly becomes intractable, and therefore most proofs of this sort end up being incorrect beyond a certain level of code complexity.

---

To summarize...

• SeqCst provides obscure guarantees that are only needed in very weird synchronization protocols whose correctness is very hard to prove.
• SeqCst obscures non-atomic data synchronization guarantees with respect to Acquire, Release and AcqRel, because as an ambiguous term it makes it less clear what kind of non-atomic writes and reads may be synchronized by it.
• As a result, many people misunderstand SeqCst as a way to get Release ordering on loads or Acquire ordering on stores, which is impossible at the hardware level. These misunderstandings lead to incorrect synchronization code.

For all these reasons, I personally consider presence of SeqCst orderings in atomics synchronization as a code smell, which suggests that the programmer who designed the synchronization protocol likely did not fully understand atomics and their memory orderings, and dropped in a SeqCst just to be safe, without having a solid understanding of whether atomics-based synchronization was applicable to the problem at hand at all.

Therefore, I would advise against suggesting SeqCst as a default atomic memory ordering in any kind of pedagogical material, and would instead advise pointing readers towards pedagogical material that actually explains when atomics-based synchronization is appropriate and which memory orderings must be used for what situation.

For example, I consider this blog post by @jeehoonkang to be an excellent starting point.