lazy_static without locks?

[pganssle]

I have been working on a wrapper of Python's datetime library, which has a one-time initialization of an API object. I felt that lazy_static gave the best semantics, but it turns out it can cause deadlocks when used in multiple threads if the stuff inside the lazy_static acquires locks.

In my example, I have:

lazy_static! {
    pub static ref PyDateTimeAPI: PyDateTime_CAPI = unsafe { PyDateTime_IMPORT() };
}

The PyDateTimeAPI is only accessed within functions that have acquired the global interpreter lock (GIL), but PyDateTime_IMPORT() releases and then re-acquires the GIL. This causes a deadlock as such:

• Thread 1 acquires GIL
• Thread 2 blocks on GIL
• Thread 1 acquires lazy_static lock
• Thread 1 calls import, which releases the GIL
• Thread 2 acquires GIL
• Thread 2 blocks on lazy_static lock
• Thread 1 completes import call and blocks on GIL

In this case, I happen to know that the code I'm calling is already thread-safe, and the value of PyDateTime_IMPORT() is guaranteed by contract to return the same object on a subsequent call.

Is it possible to get a version of lazy_static that does not attempt to acquire a lock? Having it be unsafe is also fine.

Another option might be that the no-lock version of lazy_static doesn't lock around the function body, but does lock around assigning the value. In pseudocode it would be something like this:

let mut const_val : Option<T> = None;
fn get_const_val<T, F>(f: F) -> &T {
    match const_val {
        Some(v) => v,
        None => {
                let v = f();    // No lock, this may be called many times
                global_lock.acquire();
                // If f() was called maybe once and we lost the race,
                // the previous invocation will have set const_val
                let rv = match const_val {
                    Some(cv) => cv,
                    None => {
                        const_val = v;
                        v
                    }
                };
                global_lock.release();
                rv
        }
    }
}

So long as the function body has no side effects, the worst case scenario is duplicated effort.

I'd be fine if the macro were only available in unsafe blocks.

[KodrAus]

Hi @pganssle, hmm this is pretty unfortunate, but possibly a bit out-of-scope for lazy_static to support. I think the semantic differences are a bit too subtle for us to support both approaches, and there are equal drawbacks to duplicated effort where other side-effects are involved in the initialization.

Have you already worked around this in your code?

[pganssle]

@KodrAus Yes, I took effectively the same approach lazy_static does, implementing Deref on a "dummy struct", but locking around the assignment rather than the execution of the block that returns the value I want. See here.

I totally understand if it's out of scope, it's a pretty unusual situation.

[KodrAus]

@pganssle Glad to hear you found your way around it :+1:

Something I think we could do better is describe how values are initialized in the docs better. We have this section on implementation details, but it's pretty light in details.