Supervisor leaks memory when fibers are canceled quickly

[TomasMikula]

Reproduction

//> using scala 3.7.3
//> using dep org.typelevel::cats-effect:3.6.3

import cats.effect.*
import cats.effect.std.Supervisor
import scala.concurrent.duration.*

object SupervisorTest extends IOApp:
  val M = 20
  val N = 100000

  override def run(args: List[String]): IO[ExitCode] =
    for
      _ <- reportMemory
      _ <- Supervisor[IO]
        .use: supervisor =>
          for
            _ <- supervisor
              .supervise(IO.unit)
              .flatMap(_.cancel) // Note: join would be leak free
              .replicateA_(N)
              .parReplicateA_(M)
            _ <- IO.sleep(5.seconds)
            _ <- reportMemory
          yield
            ()
    yield
        ExitCode.Success

  private def reportMemory: IO[Unit] =
    IO.delay:
      val runtime = Runtime.getRuntime()
      runtime.gc()
      val allocatedMemory = runtime.totalMemory() - runtime.freeMemory()
      val allocatedMB = allocatedMemory / (1024 * 1024)
      println(s"Memory taken: $allocatedMB MB")

Output

% scala SupervisorTest.scala
Compiling project (Scala 3.7.3, JVM (21))
Compiled project (Scala 3.7.3, JVM (21))
Memory taken: 4 MB
Memory taken: 510 MB

That is, after the supervisor has started a bunch a bunch of fibers and all of them are canceled, the memory consumption is 500+ MBs higher than before.

Notes

If the _.cancel is changed to _.join, the final memory consumption remains at 4 MB.

You can vary the value of N to vary the amount of leaked memory.

This was first suspected in #4488, as the implementation of Supervisor expects that .start guarantees execution of finalizers added with .guarantee, .guaranteeCase.

[durban]

See #4500.