Consider adding RAII

[MarvinHannott]

I would like to propose new semantics for RAII in Dart.

For a class to be a resource, it should be annotated as resource, which requires the dispose method. For a variable to hold a resource, it must be annotaded with using. This is necessary for two reasons: The binding needs to be final; binding a new value to the variable would cause a resource leak. The second reason is that the resource cannot escape its scope. That means the reference (after all, classes are reference types) cannot be copied to a variable outside the resource's scope. Copying the reference is only legal within the scope. At the end of the scope, all resources should get destroyed in reverse order and the dispose methods should be called.

resource SomeResource {
  // other methods

  void _dispose() {
    // some cleanup
  }
}

void usingResource() {
  using resource = new SomeResource();
  {
    using anotherResource = new SomeResource();
  }
  // anotherResource gets destroyed
}
// resource gets destroyed

Of course there are cases where a resource must escape its scope, for example when it gets returned by a function. In this case the responsability of disposing the resource shifts from the function to the caller:

SomeResource returningResource() {
       using resource = new SomeResource();
       return resource;
}

// caller
void main() {
      using resource = returningResource();
}
// resource's dispose method gets called

On the caller's side the same rules apply: He needs to annotate the receiving variable with using and when it hits the end of the scope it gets destroyed. Resources which are function arguments of course outlive the function's scope and cannot be destroyed within the function.

Another case where a resource needs to escape is when it is part of another resource. Resources can only be part of other resources! They cannot be part of other classes because that would mean resource leaks. When the encapsulating resource gets destroyed, all encapsulated resources get destroyed as well, even without explicitly calling their dispose method. This ensures that encapsulated resources will get destroyed (at the latest) when the encapsulating resource gets destroyed itself. But it can also be done manually.

This of course raises the question what to do when a resource ist part of several other resources. Two possible solutions would be reference counting and weak references. Which is the right one depends on the use case.

I hope this is helpful and sparks new ideas.

[edward-a]

'using' statement also insures that the resource gets disposed in case of exception. Similar to how 'finally' statement works.

[ryanhaining]

I definitely support the core idea of deterministic resource release, though some of the suggested restrictions (resources can only be part of other resources) seem too strict.

If all we're interested in is release, I would suggest borrowing a bit from python and a bit from Java/C# here. Create a disposable (or autocloseable, or whatever) interface

class MyResource implements Disposable {
  @override
  void dispose { /* ... */ }
}

Syntactically, use a similar syntax to Python context managers.

with (var resource = MyResource()) {
  // ...
}

However Python's context managers actually use __enter__ at the start of the block, and __exit__ at the end, which means you can construct the object separately, and reuse it in various with blocks. This makes sense to me since the memory model is a lot more similar than C++ (raii). For locking a mutex in a critical section

with mutex:
    critical_section()

I know a mutex doesn't mean much in dart, but hopefully this is still a relatable example. Probably using different names, it would be possible to do something similarly useful:

class Mutex implements Manageable {
  @override
  void enter() {
    this.acquire();
  }
  
  @override
  void exit() {
    this.release();
  }
}

Then use like so

var m = Mutex();
with (m) {
  criticalSection();
}

[MarvinHannott]

I definitely support the core idea of deterministic resource release, though some of the suggested restrictions (resources can only be part of other resources) seem too strict.

But I think the so called disposable pattern is more or less the same thing: The nested object's dispose method has to be called in the classes own dispose method to release the resource. Maybe that could be a Dartanalyzer rule.

If all we're interested in is release, I would suggest borrowing a bit from python and a bit from Java/C# here.

I actually looked at Hack: https://docs.hhvm.com/hack/statements/using And as far as I know Hack enforces the use of using and deterministic disposal. But this too could be a Dartanalyzer rule.

I don't think I am the only one who thinks C# or Java's solution is a little bit verbose. Hack's solution looks very similar to C++ or Rust. On the other hand any solution is better than none.

However Python's context managers actually use enter at the start of the block, and exit at the end, which means you can construct the object separately, and reuse it in various with blocks. his makes sense to me since the memory model is a lot more similar than C++ (raii).

I don't think many people would consider that to be a good idea. And especially from a C++ point of view I don't really see how that makes sense. After disposal the object is invalid and shouldn't be used. If the memory gets freed without waiting for GC it would even be impossible. Though most likely it would be most performant to let the GC handle it. But instead of reusing an object just create a new one.

[MarvinHannott]

I think the new FFI, which brings manual c-level memory management with it, makes the point for RAII in Dart even stronger. This kind or manual memory management leads to very subtle memory leaks and bugs, especially in conjunction with exceptions, even when using try-finally.

Pointer<Uint8> ptr1 = someCondition ? allocate() : null;
Pointer<Uint8> ptr2 = allocate();
try {
   // ...
} finally {
  ptr1.free() // Might be null
  ptr2.free() // If ptr1 is null an exception gets thrown and ptr2 will never get freed
}

This was a typical situation in Java before try-with-resource. It leads to subtle memory bugs like this or files that don't get released again. Yes, one could always do a defensive null check, but that comes at a cost and the JIT knows better if a variable could be null or not. Moreover the code above contains is another bug: What if the second allocation fails? Then the first pointer would never get freed.

[ryanhaining]

I wouldn't push too hard for __enter__ and __exit__ like functionality, it's just something to consider.

It seems like as long as you have a sane model for non-owning references (as with C++) it'd definitely be useful. If I understand your suggestion about all Resources being part of another Resource then that means no non-resource object could hold a non-owning reference to a resource. Which might be best because it means no dangling references I guess, but feels a little overkill, I haven't personally witnessed problems with people keeping references to objects closed via context managers or try-with-resources blocks.

I don't really know much about hack but from that link

to pass it to a function, we must mark the function's corresponding parameter with the attribute __AcceptDisposable

Would you expect an @annotation to do this job in dart?

[MarvinHannott]

It seems like as long as you have a sane model for non-owning references (as with C++) it'd definitely be useful.

Both Java and C#, but also Swift have Weak References. JavaScript also has WeakMap and WeakSet. I think that would be a fine solution. Might be worth investigating if something like that could be implemented via Native Extension: https://dart.dev/server/c-interop-native-extensions

If I understand your suggestion about all Resources being part of another Resource then that means no non-resource object could hold a non-owning reference to a resource. Which might be best because it means no dangling references I guess, but feels a little overkill, I haven't personally witnessed problems with people keeping references to objects closed via context managers or try-with-resources blocks.

This special Resource thing is due to the proposed semantics:

resource SomeResource {
 // other methods

 void _dispose() {
   // some cleanup
 }
}

class NestsResource {
 SoumeResource resource;
 NestsResource(this.resource);
}

NestsResource usingResource() {
 using resource = new SomeResource();
 final nested = new NestsResource(resource);
 return nested;
}
// [resource] gets disposed after return and would be in an invalid state.
// A Weak Reference would be okay and just return null.

It resembles this situation:

class SomeResource {
 // other methods

 void close() {
   // some cleanup
 }
}

class NestsResource {
 SoumeResource resource;
 NestsResource(this.resource);
}

NestsResource usingResource() {
 final resource = new SomeResource();
 try {
   final nested = new NestsResource(resource); 
   return nested;
 } finally {
   resource.close();
 }
}
// [resource] gets closed after return and would be in an invalid state.
// This could be a very subtle bug.

Of course you could just not close resource, but then the obligation to do so shifts to the caller, which is exactly what I try to avoid, especially when dealing with memory through Dart's FFI. This is exactly why C++ has Smart Pointers.

resource SomeResource {
 // other methods

 void _dispose() {
   // some cleanup
 }
}

resource NestsResource {
 SoumeResource resource;
 NestsResource(this.resource);
 void _dispose(){
   // might or might not call [resouce._dispose] directly
   // it will get triggered regardless when [this._dispose] gets called
 }
}

NestsResource usingResource() {
 using resource = new SomeResource();
 // Now [nested] is responsible for disposing [resource] because it gets an owning
 // reference to it which supresses the call of [resource._dispose] And because [nested] escapes
 // the function, its own dispose method doesn't get called at this point.
 using nested = new NestsResource(resource);
 return nested;
}
// [resource] is still in a valid state. Its dispose method will get called when [nested.dispose]
// gets called. [nested.dispose] gets called when it hits the end of the surrounding scope.

Consequently this shouldn't be allowed or at least produce a warning by Dartanalyzer:

NestsResource usingResource() {
  using resource = new SomeResource();
  using nested = new NestsResource(resource);
  using nested1 = new NestsResource(resource);
  return nested;
}
// [nested1] gets disposed after return and so does [resource]. There should be only one owning reference. 

Yes, it might feel a little overkill at first, but it actually resembles C++ or Rust RAII and should be intuitive for most.

don't really know much about hack but from that link

to pass it to a function, we must mark the function's corresponding parameter with the attribute >> __AcceptDisposable

Would you expect an @annotation to do this job in dart?

I honestly don't really understand what that annotation is good for. Resource arguments naturally outlive the function call and don't get disposed inside.

[ryanhaining]

I'm seeing more what you're saying. I expect the situation to become interesting if you want to transfer ownership into a called function, or return the object. Your first post suggests move semantics for these cases which might be fine but would need weak references. For rust-like semantics with lifetimes you'd need a lot of language support.

I think you'd need to impose a similar restriction on nesting resources to final, having them always be initialized and unmodifiable.

void usingResource() {
  using nested = NestsResource();
  {
    nested.some = SomeResource(); // scoped to this block?
  }
  // nested.some dangles
}

If you can return a resource, could you do this? This is where the reference model starts to become more obviously an issue to me.

resource NestsResources {
  using someResource = SomeResource();

  SomeResource access() => someResource; // or a normal getter
}

void main() {
  using nested = NestsResource()
  {
    using rsrc = nested.access(); // move?
  }
}

I honestly don't really understand what that annotation is good for. Resource arguments naturally outlive the function call and don't get disposed inside.

if imposing your restrictions on resources needing to be inside other resources then yes, but otherwise you could take a resource and return another object with a (non-owning) reference to it.

[MarvinHannott]

I expect the situation to become interesting if you want to transfer ownership into a called function, or return the object. Your first post suggests move semantics for these cases which might be fine but would need weak references. For rust-like semantics with lifetimes you'd need a lot of language support.

I explicitly don't want Rust like move semantics. Resources (or lets call it Disposables) are just a special kind of class and therefore reference types. They definitally outlive any function they are passed into. And any changes done to them will be visible from the outside.

The case that a Disposable argument gets returned is interesting. To be honest: I haven't thought about it and don't know what should happen. The simplest answer would be: Don't allow it. It doesn't really make sense anyway. But the more practical answer could be: In that case (and only that one) there would be two owning references, and yes, the second one could be disposed before the firstm, and yes, this could lead to bugs. But I don't think that would be a common case. I understand there have to be limits for compiler checks. Could also be a Dartanalyzer rule, I suppose.

I think you'd need to impose a similar restriction on nesting resources to final, having them always > be initialized and unmodifiable.

void usingResource() {
 using nested = NestsResource();
 {
   nested.some = SomeResource(); // scoped to this block?
 }
 // nested.some dangles
}

Well, using should impose such restrictions:

For a variable to hold a resource, it must be annotaded with using. This is necessary for two reasons: The binding needs to be final; binding a new value to the variable would cause a resource leak. The second reason is that the resource cannot escape its scope. That means the reference (after all, classes are reference types) cannot be copied to a variable outside the resource's scope.

So this example shouldn't be possible; nested.some cannot be reassigend and SomeResource couldn't be referenced outside its scope. In other words: nested.some could only be initialized by the constructor. I think this is similar to what Hack is doing. But like I said, maybe a Dartanalyzer rule is good enough. I wouldn't be a fanatic about it.

If you can return a resource, could you do this? This is where the reference model starts to become >more obviously an issue to me.

resource NestsResources {
 using someResource = SomeResource();

 SomeResource access() => someResource; // or a normal getter
}

void main() {
 using nested = NestsResource()
 {
   using rsrc = nested.access(); // move?
 }
}

I don't think there should be move semantics. This is were Weak References come into play. You don't want error messages lik Rust's famous "Cannot move out of borrowed content". So there should be one "owner" and "ownership" cannot be passed; only Weak References. The only exception is a Disposable which is "owned" by another Disposable, so only the constructors of Disposables have move semantics. I was indeed very unclear in that regard, but I think that would be a reasonable choice. Sprinkling Rust like move semantics or even borrowing would be really out of place.

I honestly don't really understand what that annotation is good for. Resource arguments naturally >>outlive the function call and don't get disposed inside.

if imposing your restrictions on resources needing to be inside other resources then yes, but otherwise you could take a resource and return another object with a (non-owning) reference to it.

I am not sure I understand the problem. Yes, one could return a object with a Weak Reference to a Disposable argument. Or do you mean that a Disposable gets created inside a function and then a Weak Reference to that Disposable gets returned? Well, I don't think there is anything that could stop anybody from doing that (maybe another Dartanalyzer rule 😉).

[ryanhaining]

The reason I'm distinguishing between weak references and non-owning ones here is that a non-owning reference would prevent garbage collection, but not disposal. If that's what you're talking about then this all seems like a pretty solid idea to me.

What I meant with the last point was something like

resource SomeResource {
  // ...
}

class Refers {
  NonOwning<SomeResource> resource;
  Refers(this.resource);
}

Refers f() {
  using resource = SomeResource();
  var refers = Refers(resource.nonOwning());
  return refers;
}

tbc, this is totally doable in Python, Java, or (similarly) C++. I suppose the question would be: how often would people want objects with non-owning references to resources?

[MarvinHannott]

The reason I'm distinguishing between weak references and non-owning ones here is that a non-owning reference would prevent garbage collection, but not disposal. If that's what you're talking about then this all seems like a pretty solid idea to me.

Ah, I see. No, I always had weak references in mind which wouldn't prevent gc. When an object is in an invalid state after disposal, there wouldn't be any reason to prevent gc. The way I see it, the memory could even be freed at once, it just might not be the most efficient way. I don't think disposed objects should be available anymore like they aren't in C++ or Rust after destruction.

But it might very well be the case that I am overlooking something and there actually were good reasons for non-owning references which prevent gc even after disposal. My solution would be to just copy the bits of information I still need. It would also still be possible to reference non-disposable objects so they wouldn't be collected:

Object f() {
 using resource = SomeResource();
 var refers = resource.nonDisposableObject
 return refers;
}

So this works as usual.

[makoConstruct]

I think this part of the proposal was a mistake:

The second reason is that the resource cannot escape its scope. That means the reference (after all, classes are reference types) cannot be copied to a variable outside the resource's scope.

I've used Rust enough to know that restricting where a reference can be sent is seriously cumbersome, and also this is just not necessary in a high level gc context that already has auto dispose. As long as something has been disposed, it doesn't matter very much if outside contexts still have references to it, as it would just be a weak ref to a little tombstone at that point, what needed to be disposed already has been.

[makoConstruct]

I see a way of implementing automatic disposal with few or no new language constructs, mostly just annotations and codegen.

class ExampleState extends State {
  @dispose FocusNode focusNode = FocusNode();
}

@dispose would just add focusNode.dispose() to a generated implementation of @override dispose(){ focusNode.dispose(); super.dispose(); } which you get if you have any @dispose annotations.

Ideally you'd be able to override void dispose() yourself, and there'd be something like @mustCall(_disposeLocalMembers) (afaik mustCall doesn't exist) that issues a warning if you fail to call _disposeLocalMembers in your dispose implementation, and somehow _disposeLocalMembers would always refer to the _disposeLocalMembers implementation of its own class rather than the most recent override in the chain, which I'm not sure is possible in dart, but somehow super. refers to specific overrides, so that probably wouldn't be hard for the project to implement.

[lrhn]

If you are willing to do a little more work, it's fairly easy to have resources allocated in an allocation zone, and have them disposed when leaving that zone.

Not as easy as C++ RAII, but Dart does not have destructors, so the most a dispose method can do is to free internal state of an object, the object itself still exist. The stack allocation going away, like in C++, is not happening in Dart. Also Dart has asynchrony, so "end of computation" is a completely artifical concept for Dart.

(But it could technically be possible to have code run when a variable goes out of scope. It's just annoying for a garbage collected language to keep values alive until the end of a scope, if they aren't actually used in the last part of that scope.)

import "dart:async";

abstract base class Resource {
  // Do implement this.
  void dispose();

  static List<Resource>? get _currentZone {
    var zone = Zone.current[#_resourceZone];
    if (zone is List<Resource>) return zone;
    return null;
  }

  static R zone<R>(R Function() body) {
    var newZone = <Resource>[];
    R result;
    try {
      result = runZoned<R>(body, zoneValues: {#_resourceZone: newZone});
    } on Object {
      _disposeAll(newZone);
      rethrow;
    }
    if (result is Future) {
      return (result as Future<Object?>).whenComplete(() {
        _disposeAll(newZone);
      }) as R;
    }
    _disposeAll(newZone);
    return result;
  }

  static void _disposeAll(List<Resource> resources) {
    for (var i = resources.length - 1; i >= 0; i--) {
      try {
        resources[i].dispose();
      } catch (e, s) {
        Zone.current.handleUncaughtError(e, s);
      }
    }
  }

  Resource() {
    var zone = _currentZone;
    if (zone == null) {
      dispose();
      throw StateError("Not in any resource allocation zone");
    }
    zone.add(this);
  }
}

final class MyResource extends Resource {
  final String name;
  MyResource(this.name) {
    print("Allocated: $name");
  }
  void dispose() {
    print("Disposed: $name");
  }
}

extension ZoneFunction<R> on R Function() {
  R Function() get zoned {
    return () => Resource.zone(this);
  }
}

extension ZoneFunction1<R, P1> on R Function(P1) {
  R Function(P1) get zoned {
    return (P1 p1) => Resource.zone(() => this(p1));
  }
}

extension ZoneFunction2<R, P1, P2> on R Function(P1, P2) {
  R Function(P1, P2) get zoned {
    return (P1 p1, P2 p2) => Resource.zone(() => this(p1, p2));
  }
}

void main() {
  print("Starting");
  Resource.zone(doMore);
  print("Done");
}

void doSomething(MyResource s1, MyResource s2) {
  var r1 = MyResource("Banana");
  var r2 = MyResource("Apricot");
  print("Using ${r1.name} ${s1.name} and ${r2.name} ${s2.name}");
}

void doMore() {
  var r1 = MyResource("Split");
  var r2 = MyResource("Rings");
  doSomething.zoned(r1, r2);
  print("Using ${r1.name} and ${r2.name}");
}