resource annotations

[andrewrk]

trafficstars

Strap your jet pack on because we're going for a pie-in-the-sky adventure.

I propose a builtin function called @resource_acquire() which returns a u64. This is an opaque identifier which is probably just random bytes. Zig code may then store this identifier somewhere, like this:

const want_resource_tracking = @compile_var("debug_safety_on");
const ResourceIdType = if (want_resource_tracking) u64 else void;
struct HeapEntry {
    debug_id: ResourceIdType,
    // ....
}
fn malloc(inline T: type, count: usize) -> %[]T {
    if (want_resource_tracking) some_heap_entry.debug_id = @resource_acquire();
}
fn free(inline T: type, mem: []T) {
    if (want_resource_tracking) @resource_release(some_heap_entry.debug_id);
}

Now, in debug mode, Zig will add safety checks to catch the following scenarios:

• (easy) (runtime) Releasing a resource too many times.
• (easy) (runtime) Program reaches end with unreleased resources.
• (hard) (compile error) The compiler was able to determine that the resource couldn't possibly have been released, or that the resource will be released twice.
• (hard) (runtime) Track the memory location where the resource id is stored and reference count it. If references drop to 0 before it gets released, runtime error, the resource leaked. The stack trace at the location the ref count drops to 0 will be very useful.

The standard library would implement these semantics for all resources, for example file streams, network connections.

If the implementation is fast enough this could even be viable for detecting failure to flush a stream, such as stdout. When anything is written, acquire a resource, unless one is already acquired. When flush is called, release the resource, unless already released.

Depending on how advanced the safety checks are, this might not even need to be a builtin function. It could just be a standard library function.

In the event of an error we would want to print the stack trace of the resource being acquired, so we might want a @get_stack_trace() builtin function. We want that anyway. Separate issue.

[andrewrk]

Better yet the void/u64 thing can be done by the compiler, and we can have a resource type.

Then you don't even need the if (want_resource_tracking) code, since assigning void does nothing.

[thejoshwolfe]

The name resource seems kind of like the name data or object. Sorry to be the one who criticizes names first all the time, but I think we can do better. My best proposal is LifeCycleTracker. This doesn't say that it's a resource, because it really isn't. It's meant to accompany a resource in order to track its life cycle. And now I think it's more believable that a life cycle tracker would be void in release mode, as opposed to a resource, which you would probably still want in release mode.

I chose the name Tracker instead of Instead of Id, because I think reference counting on something called an ID would be surprising. Tracker sounds fancy enough that it's more plausible that it could be doing reference counting. And if we want to do reference counting, it might be nice to give users direct access to it via @ref(tracker), @unref(tracker), and even @set_ref_counting_enabled(tracker, false).

If the (debug) runtime is tracking a stacktrace for where the resource was acquired, how about providing access to that? @get_acquisition_stack_trace(tracker).

Anyway, those are my loop-d-loops and barrel rolls on this jetpack ride :D

[kyle-github]

It seems like this is just a few steps from a built in reference count implementation. If you have it internally why not expose it so that programs can use it?

To thejoshwolfe, reference counting on anything can be useful in some circumstances. It is usually associated with memory, but can be file handles, sockets etc. It is not a panacea, but it really gets useful when you have a lot of async code or lots of threading. Really anything that makes the order in which resources could be freed complicated.

[lerno]

Whatever happened to this feature?

[mlugg]

It's a non-accepted proposal, which means it may or may not make it into the language. More to the point, the current milestone of 1.1.0 suggests that if this is eventually accepted, that might not be until after Zig 1.0.

[lerno]

I'm curious whether there were any further discussion on this or if it's likely DOA

[matu3ba]

This sounds pretty much like valgrind except that valgrind has in-build Kernel ressource annotations, whereas this annotation is user-based (possibly inside libstd).

Open questions:

• 1. usage: Are these resource annotations purely intended for Zig user code or how should Zig users handle/annotate externally linked object code or reused C code?
• 2. implementation: What should the compiler then emit and track exactly? Often state is implicit, meaning that resource allocation and deallocation depends on the order of execution without explicit variables. Take defer as most prominent example.
• 3. implementation: How should this work for multiple threads? Making the u64 atomic?
• 4. why: What specific advantage for eliminating the problem does this provide? Mainstream Kernels already provide tracing capabilities and there is already tooling (valgrind and strace being most commonly known ones), so why should Zig users repeat this work?
• 5. composability: How can the user pin ownership to specific runtime execution contexts (Thread, Process, functions etc) or does the user need to build their own redundant tooling for this from scratch?
• 6. composability: This does not play very nice with swappable things (for example switching allocators) that need no free (Arena, FixedBuffer). Or does it?
• 7. usage: It needs extra handling for expected panic checks and/or test behavior must annotate how the functionality should work.

From my point of view, the MVP could be done once supported and non-supported use cases have been defined with the actual advantages to reusage of current tooling including generality, composability.