Copying logfiles by `MaxSize.rollPaths` is slow

[kpodsiad]

Hey! Thanks for the great lib and the continuous effort with maintaining it.

I want to ask about https://github.com/outr/scribe/blob/e227636d7d88571d3cf4c850401281e886264bff/fileModule/shared/src/main/scala/scribe/file/path/MaxSize.scala#L41

When there are bunch of already existing log files with a lot of logs, it takes significant amount of time for the MaxSize to copy all the files. As far as I can see, it will move N -> N+1, N-1 -> N, ..., current file -> 1. Copying the files is blocking operation that blocks the execution and causes program to hang for some amount of time.

In my rudimentary example, there is only maxSize used. Perhaps using day or hour could help, but nevertheless, the underlying issue would still be present.

reproduction snippet

I've used dd if=/dev/zero of=logs/app.log bs=1M count=1024 to mimic already existing log files. Second log stops the execution for the couple of seconds needed to copy the logfiles:

it took PT0.078805S to log message in iteration 0
it took PT7.695495S to log message in iteration 1
it took PT0.031225S to log message in iteration 2
it took PT0.020617S to log message in iteration 3
it took PT0.016023S to log message in iteration 4

//> using scala "3.7.2"
//> using dep "com.outr::scribe::3.17.0"
//> using dep "com.outr::scribe-file::3.17.0"

import scribe.Logger
import scribe.format.*
import scribe.file.*
import scribe.file.path.MaxSize
import java.time.Instant
import java.time.Duration
import scala.sys.process.*
import java.io.File

object Main extends App {
  val size = MaxSize.OneHundredMeg * 5
  val file = "app" % maxSize(max = size, separator = ".") % ".log"

  val fileWriter = FileWriter("logs" / file)
  Logger.root
    .clearHandlers()
    .withHandler(writer = fileWriter)
    .replace()

  // emulating already existing logfiles
  new File("logs/").mkdirs()
  val _ = "dd if=/dev/zero of=logs/app.log bs=1M count=1024".!!
  (1 to 10).foreach { i =>
    val _ = s"dd if=/dev/zero of=logs/app.$i.log bs=1M count=1024".!!
  }

  var i = 0
  val payload = "0" * 1024 * 1024 + '\n'
  while (i < 5) {
    val before = Instant.now()
    scribe.info(payload)
    val after = Instant.now()
    val between = Duration.between(before, after)
    println(s"it took ${(between)} to log message in iteration $i")
    i += 1
  }

}

[darkfrog26]

It's been a while since I've looked at this code, but I'm open to some improvements. In fact, what makes more sense is to move the files instead of copying them, which, under nearly every FS, would incur almost no performance cost.

If you'd be interested in putting together a PR, I'd be happy to work with you to get it merged and published.