[Proposal]: Lock statement pattern (VS 17.10, .NET 9)

[stephentoub]

Lock statement pattern

• [x] Proposed
• [ ] Prototype: Not Started
• [x] Implementation: https://github.com/dotnet/roslyn/pull/71716
• [x] Specification: https://github.com/dotnet/csharplang/blob/main/proposals/lock-object.md

(This proposal comes from @kouvel. I've populated this issue primarily with text he wrote in a separate document and augmented it with a few more details.)

Summary

Enable types to define custom behaviors for entering and exiting a lock when an instance of the type is used with the C# “lock” keyword.

Motivation

.NET 9 is likely to introduce a new dedicated System.Threading.Lock type. Along with other custom locks, the presence of the lock keyword in C# might lead developers to think they can use it in conjunction with this new type, but doing so won't actually lock according to the semantics of the lock type and would instead treat it as any arbitrary object for use with Monitor.

Detailed design

Example

A type would expose the following to match the proposed pattern:

class Lock : ILockPattern
{
    public Scope EnterLockScope();

    public ref struct Scope
    {
        public void Dispose();
    }
}

public interface ILockPattern { }

EnterLockScope() would enter the lock and Dispose() would exit the lock. The behaviors of entering and exiting the lock are defined by the type.

The ILockPattern interface is a marker interface that indicates that usage of values of this type with the lock keyword would override the normal code generation for arbitrary objects. Instead, the compiler would lower the lock to use the lock pattern, e.g.:

class MyDataStructure
{
    private readonly Lock _lock = new();

    void Foo()
    {
        lock (_lock)
        {
            // do something
        }
    }
}

would be lowered to the equivalent of:

class MyDataStructure
{
    private readonly Lock _lock = new();

    void Foo()
    {
        using (_lock.EnterLockScope())
        {
            // do something
        }
    }
}

Lock pattern and behavior details

Consider a type L (Lock in this example) that may be used with the lock keyword. If L matches the lock pattern, it would meet all of the following criteria:

• L implements interface ILockPattern
• L has an accessible instance method S EnterLockScope() that returns a value of type S (Lock.Scope in this example). The method must be at least as visible as L. Extension methods don't qualify for the pattern.
• A value of type S qualifies for use with the using keyword

A marker interface ILockPattern is used to opt into the behaviors below, including through inheritance, and so that S may be defined by the user (for instance, as a ref struct). For a type L that implements interface ILockPattern:

• If L does not fully match the lock pattern, it would result in an error
• If a value of type S may not be used in the context, it would result in an error
• If a value of type L is implicitly or explicitly casted to another type that does not match the lock pattern (including generic types), it would result in a warning
  • This includes implicit casts of this to a base type when calling an inherited method
  • This is intended to prevent accidental usage of Monitor with values of type L that may be masked under a different type and used with the lock keyword, such as with casts to base types, interfaces, etc.
• If a value of type L is used with the lock keyword, or a value of type S is used with the using keyword, and the block contains an await, it would result in an error
  • The warning would only be issued if in-method analysis can detect it
  • This is intended to prevent the enter and exit from occurring on different threads. Monitor and Lock have thread affinity.
  • SpinLock is optionally thread-affinitized. It can opt into the pattern, but then usage of it with the lock or using keywords would still disallow awaits inside the block statically.

SpinLock example

System.Threading.SpinLock (a struct) could expose such a holder:

struct SpinLock : ILockPattern
{
    [UnscopedRef]
    public Scope EnterLockScope();

    public ref struct Scope
    {
        public void Dispose();
    }
}

and then similarly be usable with lock whereas today as a struct it's not. When a variable of a struct type that matches the lock pattern is used with the lock keyword, it would be used by reference. Note the reference to SpinLock in the previous section.

Drawbacks

• If an existing reference type opted-in to using the pattern, it would change the meaning of lock(objectsOfThatType) in existing code, and in ways that might go unnoticed due to the general unobservability of locks (when things are working). This also means if System.Threading.Lock is introduced before the language support, it likely couldn't later participate.
  • Also if one library L1 that exposes the type is updated to opt into the lock pattern, another library L2 that references L1 and was compiled for the previous version of L1 would have different locking behavior until it is recompiled with the updated version of L1.
• The same thing is achievable today with only a little bit more code, using using instead of lock and writing out the name of the enter method manually (i.e. writing out the lowered code in the first example).
• Having the lock semantics vary based on the static type of the variable is brittle.
• ILockPattern itself doesn't match the lock pattern, though an interface could be created to match the lock pattern.

Alternatives

• Use an attribute instead of a marker interface. A marker interface was used because it works more naturally with inheritence. Although AttributeUsageAttribute.Inherited = true could be used for an attribute, it doesn't seem to work with interfaces.
• Use a new syntax like using lock(x) or other alternatives instead, which would eliminate any possibility of misusing values of the type with Monitor.
• System.Threading.Lock could be special-cased by the compiler rather than having it be a general pattern.
• Various naming options in the pattern itself.
• Instead of having a holder type, instead have the compiler explicitly recognize certain enter/exit methods by naming convention or attribution.
• Potentially allow/require the ref keyword for certain uses, e.g. in the SpinLock example.
• The pattern as suggested doesn't support being hardened against thread aborts (which are themselves obsoleted). It could be extended to now or in the future.

Unresolved questions

Design meetings

https://github.com/dotnet/csharplang/blob/main/meetings/2023/LDM-2023-05-01.md#lock-statement-improvements https://github.com/dotnet/csharplang/blob/main/meetings/2023/LDM-2023-10-16.md#lock-statement-pattern https://github.com/dotnet/csharplang/blob/main/meetings/2023/LDM-2023-12-04.md#lock-statement-pattern

[tfenise]

1. If a System.Threading.Lock lo becomes typed as Object at compile-time for whatever reason, lock(lo) would break silently. For example:

Dictionary<string, object> dic = new();
Lock lo = new();
lock(dic["my lock"] = lo) {}//this does not work.

2. This pattern match design creates potential problems for codes which implement custom locks and want to take advantage of this new language feature. For example, suppose I implement my own lock:

class MyLock
{
    public LockHolder EntersLockWithHolder() {...}

    public struct LockHolder
    {
        public void Dispose() {...}
    }
}

...

static readonly MyLock lo = new();

static void MyMethod()
{
    lock(lo)
    {
        ...
    }
}

Everything compiles even though the spelling EntersLockWithHolder is wrong and lock(lo) does not do what I intended.

This may also be problematic if MyLock is first written against .NET 8 and is later ported to .NET 7 or .NET Framework for whatever reason. In these cases, the codes would also break silently.

This problem could be solved by introducing a custom attribute only to be found in .NET 8 BCL that must be present on such lock types. The compiler would then detect any misspelling, and backporting to earlier version would result in a compilation error.

3. This new design creates inconsistency between newer lock types that should be used with lock(...) and older lock types like Mutex that should not be used with lock(...).

4. If this new design is implemented in C#, it should also be implemented in VB.NET, as VB.NET has SyncLock which is basically equivalent to lock of C#.

[vladd]

If we consider implementing this feature, shouldn’t we consider its async counterpart which is lowered to await using, too?

[HaloFour]

@vladd

If we consider implementing this feature, shouldn’t we consider its async counterpart which is lowered to await using, too?

I believe that should follow a completely different pattern as locks should have to opt-in to being "async"-friendly.

[HaloFour]

@tfenise

If a System.Threading.Lock lo becomes typed as Object at compile-time for whatever reason, lock(lo) would break silently.

I have to agree, I think there's enough danger of these newer-style locks being accidentally treated as monitor locks by the compiler in any case where the pattern just happens to not match, with no warning to the developer. I think that needs to be added to the list of "Drawbacks" for this proposal.

Is the plan to also extend this to the myriad of other locks or synchronization primitives that exist in the BCL?

[svick]

Would it be feasible to introduce the ability of a type to forbid using Monitor on it into the runtime? Let's say this feature would be enabled by annotating the type as [ForbidMonitor]. Then the new Lock could look like this:

[ForbidMonitor]
class Lock
{
    public LockHolder EnterLockWithHolder();

    public struct LockHolder
    {
        public void Dispose();
    }
}

And used like this:

private readonly Lock _lock = new();

void Foo()
{
    lock (_lock) // lowers to EnterLockWithHolder()
    {
    }

    lock ((object)_lock) // lowers to Monitor.Enter, but throws at runtime
    {
    }
}

[HaloFour]

@svick

How would that be enforced? If you upcast the Lock it's just a normal object and the compiler wouldn't know to not use Monitor.

[alexrp]

@HaloFour As per the comment, it would be runtime-enforced. Presumably, if [ForbidMonitor] is applied to the type, some bit in the method table would be set to indicate that any Monitor methods called on the object should throw.

[svick]

@HaloFour I meant that the compiler would use Monitor, but the implementation of Monitor.Enter would detect that the type of the object is one of the annotated ones and would throw an exception.

But I don't know if that could be implemented without making every existing use of lock slower.

[HaloFour]

@alexrp @svick

Sorry, I see what you meant after another re-reading of the comment. It'd prevent a subtle break, but you could still end up with exceptions at runtime if you're not careful.

I'd love to see more language support around locking types. It's always annoying me that none of the lock types in the BCL even offer support for using, and I've written a bunch of extension methods specifically to enable that. It's nice to see that they're considering it for this new Lock class. I, personally, think that's sufficient. The problem with lock is that it's always going fallback to Monitor, and unless we want to make generic support for these locks a part of the runtime itself it feels a bit too dangerous to me to try to expand on that, at least without some novel guardrails. Honestly, I wish .NET could deprecate Monitor in general.

[timcassell]

@vladd

If we consider implementing this feature, shouldn’t we consider its async counterpart which is lowered to await using, too?

I believe that should follow a completely different pattern as locks should have to opt-in to being "async"-friendly.

Perhaps await lock?

Also, though I have seen await using used for an async lock before, I don't think it makes much sense. The popular AsyncEx library just uses using. So await lock (_lock) would lower to using (await _lock.EnterLockWithHolderAsync()).

[TahirAhmadov]

Hmm, my initial reaction is, not only we shouldn't add to lock() {} in C#, but we should discourage its use, and instead recommend the using(...) {} approach, even for the Monitor-based locking. This seems to be too much of a specific type of feature to be part of the language. And I also don't like the lock suddenly doing different things depending on the type's surface area. However, I don't think the Monitor locking is bad idea by itself, using something like using(Monitor.Lock(obj)) { ... } is perfectly fine with me and I use lock currently in many places, it's just that it's really not a language-level feature in my opinion. PS. To clarify, from a clean slate perspective, I wouldn't have made all .NET objects "lockable" with Monitor; it should be a feature offered by a special type, similar to Mutex/etc. But given the legacy, the Monitor just isn't going anywhere any time soon.

[TahirAhmadov]

Having thought about it, I would have liked in a "perfect world" to have lock (and await lock) statement in C#, which would do something similar to what the OP proposes, as syntactic sugar, and then a few different types could be offered in the BCL to address various common usage scenarios. However, given how lock is now tightly related to Monitor, it's not a good idea to expand it. Perhaps, we can introduce a new keyword here? Like synch and await synch? Then lock and Monitor can both be obsoleted over time with suggestions/warnings, auto-fixers, etc.

[jaredpar]

If a System.Threading.Lock lo becomes typed as Object at compile-time for whatever reason, lock(lo) would break silently. For example:

Correct and that is an explicit design choice. This proposal breaks compat for any type which is being locked on today that just happens to fit the new locking pattern. Casting to object in a lock is the mechanism for restoring the existing behavior. Essentially, it's a key part of the proposal that it be allowed and have this behavior.

lock(dic["my lock"] = lo) {}//this does not work.

This particular case is not very realistic to me. In practice, it's very uncommon for the expression inside a lock to be anything other than a variable (local or field) so the chance of this type of problem feels very low. I think the more realistic case is a Lock instance getting converted to object as a part of a more vanilla call and the receiver later decides to lock on it. Even that seems fairly unlikely cause locking on instances you don't own is a dangerous behavior.

[HaloFour]

@jaredpar

This proposal breaks compat for any type which is being locked on today that just happens to fit the new locking pattern. Casting to object in a lock is the mechanism for restoring the existing behavior.

Sounds like something which should be much more explicit. As described, it's one accidental behavior that has an accidental fallback; a pit of failure in a pit of failure.

[omariom]

Casting to object in a lock is the mechanism for restoring the existing behavior.

Wouldn't it go against the principle of least surprise?

[CyrusNajmabadi]

tbh, i would expect people to be more surprised by how things work today. Intending 'lock' to have the semantics appropraite for the lock-instance they're working with, rather than it having Monitor semantics.

[TahirAhmadov]

tbh, i would expect people to be more surprised by how things work today. Intending 'lock' to have the semantics appropraite for the lock-instance they're working with, rather than it having Monitor semantics.

For a c# programmer who knows all about await, foreach, using, and their surface area semantics, but absolutely nothing about lock, it would indeed be extremely weird to realize lock is so different from these other constructs. But that's a very small (if at all existent) demographic. Most people are very much used to lock relying on Monitor and this is a major departure. Having said that, it is weird how lock works currently, which is why I think a new keyword and an orderly, safe transition is in order.

[jaredpar]

Sounds like something which should be much more explicit

As I mentioned the far more common case of locking is done against a variable: a local or field.

lock (guard) { 

In that case the compat fix, casting to object, is very explicit and easy to identify

lock ((object)guard)

As described, it's one accidental behavior that has an accidental fallback; a pit of failure in a pit of failure.

I don't agree this is an accidental fallback, it's a very deliberate one. This new behavior is entirely about API matching. Code that wants to take advantage of that needs to be careful to maintain the correct type.

The alternative is to essentially provide a diagnostic whenever a lockable type is cast to a base type or interface. That seems quite extreme and prevents these types from having other useful behaviors. Customers who want to use the new locking pattern and are warry of this problem can simply implement it on a ref struct.

[HaloFour]

In that case the compat fix, casting to object, is very explicit and easy to identify

Except when the cast (or generic code) is far removed from the lock keyword. That's where I have a problem with it, an accidental case of falling back to Monitor.

Admittedly, as with Monitor-based locks, I'd think it'd be somewhat uncommon for these instances to be passed around, so maybe the concern is somewhat minimal.

The alternative is to essentially provide a diagnostic whenever a lockable type is cast to a base type or interface.

If the fallback is explicitly intended to be used as described then the diagnostic would only ever apply when passing the Lock instance somewhere it probably shouldn't go, which sounds like a good use case for an analyzer anyway. For anyone who wants those "other useful behaviors", they can silence the analyzer.

Customers who want to use the new locking pattern and are warry of this problem can simply implement it on a ref struct.

Customers aren't writing their own lock types, they're relying on the myriad of types provided by the BCL, none of which are ref struct or have been deemed important enough to warrant language support. It'll be really confusing as to why lock works with System.Threading.Lock but doesn't work with System.Threading.ReaderWriterLock or any of the other lock types, nor could any of those locks be "updated" without breaking a ton of existing code.

[timcassell]

It'll be really confusing as to why lock works with System.Threading.Lock but doesn't work with System.Threading.ReaderWriterLock or any of the other lock types, nor could any of those locks be "updated" without breaking a ton of existing code.

I think existing locks could be "updated" by providing an extension that returns a light wrapper that implements the new API. That would be required to work with ReaderWriterLock anyway.

lock (rwl.ReaderLock()) { }
lock (rwl.WriterLock()) { }
lock (rwl.UpgradeableReaderLock()) { }

And perhaps even do the same thing with object, and add an analyzer suggestion to convert existing locks to the extension:

lock (obj.MonitorLock()) { }

[HaloFour]

@timcassell

I think existing locks could be "updated" by providing an extension that returns a light wrapper that implements the new API.

Yes, although that suffers from the problem in that if you forget to bring that extension method into scope you'll accidentally fallback to Monitor again. Feels like you'd want analyzers that would also detect Monitor locks against common lock classes to warn of such.

And perhaps even do the same thing with object, and add an analyzer suggestion to convert existing locks to the extension:

I kind of wish we could deprecate the current behavior with lock and also put it behind this pattern, but even if that could be done with zero overhead I kind of doubt that there would be any appetite to do it.

[timcassell]

Yes, although that suffers from the problem in that if you forget to bring that extension method into scope you'll accidentally fallback to Monitor again.

Just add them directly as members to those types, then. The analyzer should warn against any lock that uses the old style Monitor, whether those are implemented as extensions or type members either way.

Feels like you'd want analyzers that would also detect Monitor locks against common lock classes to warn of such.

Absolutely! Just do it with any type that implements the lock pattern (or contains an instance method or extension that returns a type that implements the lock pattern).

And perhaps even do the same thing with object, and add an analyzer suggestion to convert existing locks to the extension:

I kind of wish we could deprecate the current behavior with lock and also put it behind this pattern, but even if that could be done with zero overhead I kind of doubt that there would be any appetite to do it.

Why not? It seems like most people here are worried about compatibility issues, not that they don't want the pattern updated. I think an analyzer warning should fix that. Even if a System.Threading.Lock instance is passed around as a System.Object, with the new pattern, any lock (state) { } will warn, so you must explicitly state how you want the lock to behave.

[HaloFour]

@timcassell

Just add them directly as members to those types, then.

That'd be up to the BCL team, and that would potentially change the meaning of existing code.

Absolutely! Just do it with any type that implements the lock pattern.

If the type already implemented the lock pattern then the fallback to Monitor would no longer be a risk.

Why not? It seems like most people here are worried about compatibility issues, not that they don't want the pattern updated.

Right, that due to the existing Monitor behavior that any accidental miss of the lock pattern would still be valid code that would compile, leading to subtle bugs at runtime. I'd be game for a more explicit approach, such as the one that you're describing, which would discourage the current behavior.

[timcassell]

Another thing, could the lock pattern be updated to be able to use the returned disposable? Like this with a using statement:

using (var key = _lock.EnterLockWithHolder())
{
    // do something with key
}

Maybe something like this?

lock (_lock; var key)
{
    // do something with key
}

[jaredpar]

If the fallback is explicitly intended to be used as described then the diagnostic would only ever apply when passing the Lock instance somewhere it probably shouldn't go, ...

Disagree. As I mentioned, this diagnostic would fire when you cast a lock instance into an interface it implemented. I would not classify that as somewhere it probably shouldn't go. The author implemented the interface, it should be a valid target. Further such a diagnostic can't even be completely accurate. It won't catch cases like the following:

object SneakyCast<T>(T t) => (object)t;

Customers aren't writing their own lock types, they're relying on the myriad of types provided by the BCL, none of which are ref struct or have been deemed important enough to warrant language support.

It's fairly trivial to wrap the BCL types in a ref struct should a customer want that level of protection

[HaloFour]

@jaredpar

It's fairly trivial to wrap the BCL types in a ref struct should a customer want that level of protection

The customers sophisticated enough to know to do that are not the customers I would be worried about.

[TahirAhmadov]

I think even if the incident rate of misusing lock is low, the fact that we're going to have one statement do two different things, is going to prove confusing over time. When you read code, every lock can be one of two things, unless you inspect the type of the expression.

[timcassell]

@jaredpar Why would you warn on a cast? Just warn when the lock statement is used on an object that doesn't implement the lock pattern.

I think even if the incident rate of misusing lock is low, the fact that we're going to have one statement do two different things, is going to prove confusing over time. When you read code, every lock can be one of two things, unless you inspect the type of the expression.

@TahirAhmadov I would be fine with a new keyword. But lock is such a nice keyword. 😞

[Joe4evr]

I suggested this in the thread about the Lock type but what if pattern-based locking is done with using lock? This would be an error if the supplied object doesn't match the required shape (good diagnostic that the object in question doesn't support it, instead of silently falling back to Monitor), and Monitor-based locking is as simple as leaving off the using.

[timcassell]

I suggested this in the thread about the Lock type but what if pattern-based locking is done with using lock? This would be an error if the supplied object doesn't match the required shape (good diagnostic that the object in question doesn't support it, instead of silently falling back to Monitor), and Monitor-based locking is as simple as leaving off the using.

That's an interesting idea, but I think that leads to another pit of failure if one forgets to add using before lock. I think warning on existing lock usage is clearer (even if we still use using lock and just deprecate lock entirely).