High memory use in parallel

[karldw]

Hello,

I'm using terra to calculate statistics for many small raster files -- I read the file, do a few manipulations, and return a small vector of numbers. I thought this workflow should fit into standard parallelization schemes, since I can pass the filename to the sub-process (avoiding issues in #36 with terra objects being un-serializable), and am returning very small objects. However, I find that using terra in parallel uses a lot of memory, even for the small elev.tif file. The example below, with 500 iterations, is small compared to my real use case. I find I'm running out of RAM when I use many thousands of files.

Does this pattern of memory use seem correct to you? It's certainly possible I'm making mistakes in my parallelization. If you think this is user error rather than a issue with terra, just let me know and I can ask stackoverflow instead.

A couple of notes about the example below:

• It takes about 45 seconds to run on my machine.
• In this example I use getFreeMemoryKB to get the system's available memory, so this code is sensitive to other things that happen on the computer.
• I included a placebo that creates a matrix in each iteration. It's simpler than terra operations, but provides a bit of a benchmark.
• Required packages are "terra", "furrr", "future.callr" and their dependencies.
• A major difference between multisession and callr is multisession uses persistent processes while callr uses a new process for each item. I think this explains the difference in "after" memory use below.

max_elev <- function(idx) {
  f <- system.file("ex/elev.tif", package="terra")
  elev_rast <- terra::rast(f)
  max <- max(terra::values(elev_rast))
  # Don't care about the elev value; return current mem free
  if (idx %% 100 == 0) {
    mem <- getFreeMemoryKB()
  } else {
    mem <- NA_integer_
  }
  mem
}


placebo <- function(idx) {
  # matrix with the same dimension as elev.tif
  m <- matrix(runif(90 * 95), nrow=90)
  max <- max(m)
  if (idx %% 100 == 0) {
    mem <- getFreeMemoryKB()
  } else {
    mem <- NA_integer_
  }
  mem
}


getFreeMemoryKB <- function() {
  # Thanks stackoverflow! (CC BY-SA 4.0) https://stackoverflow.com/a/58216766
  gc()
  Sys.sleep(0.5) # allow time for mem to be released
  osName <- Sys.info()[["sysname"]]
  if (osName == "Windows") {
    x <- system2("wmic", args =  "OS get FreePhysicalMemory /Value", stdout = TRUE)
    x <- x[grepl("FreePhysicalMemory", x)]
    x <- gsub("FreePhysicalMemory=", "", x, fixed = TRUE)
    x <- gsub("\r", "", x, fixed = TRUE)
    return(as.integer(x))
  } else if (osName == 'Linux') {
    x <- system2('free', args='-k', stdout=TRUE)
    x <- strsplit(x[2], " +")[[1]][4]
    return(as.integer(x))
  } else {
    stop("Only supported on Windows and Linux")
  }
}


mem_message <- function(msg_part, free_before, free_during, free_after) {
  free_during <- min(free_during, na.rm = TRUE)
  # Negative here to switch from diff in free mem to diff in used mem
  diff_after <- -(free_after - free_before) %/% 1024
  diff_during <- -(free_during - free_before) %/% 1024
  
  message(
    msg_part, " mem use diff: ",
    sprintf("%+d", diff_during), " MB during, ",
    sprintf("%+d", diff_after), " MB after"
  )
  invisible()
}


run_furrr <- function() {
  iter_count <- 500
  furrr::future_map_dbl(
    seq_len(iter_count),
    max_elev, 
    .options = furrr::furrr_options(seed=TRUE, globals=c("read_elev_once", "getFreeMemoryKB"))
  )
}


run_placebo <- function() {
  iter_count <- 500
  furrr::future_map_dbl(
    seq_len(iter_count),
    placebo, 
    .options = furrr::furrr_options(seed=TRUE, globals=c("read_elev_once", "getFreeMemoryKB"))
  )
}


# Run once first
max_elev(1)
placebo(1)

message("This takes a minute, but please don't do anything else with your computer until it's done.")
future::plan("sequential")

before <- getFreeMemoryKB()
during <- run_furrr()
after <- getFreeMemoryKB()

mem_message("sequential terra    ", before, during, after)


before <- getFreeMemoryKB()
during <- run_placebo()
after <- getFreeMemoryKB()

mem_message("sequential placebo  ", before, during, after)


future::plan("multisession", workers=2)

before <- getFreeMemoryKB()
during <- run_furrr()
after <- getFreeMemoryKB()

mem_message("multisession terra  ", before, during, after)

before <- getFreeMemoryKB()
during <- run_placebo()
after <- getFreeMemoryKB()

# Reset to sequential to kill the multisession processes
future::plan("sequential")
future::plan("multisession", workers=2)
mem_message("multisession placebo", before, during, after)


future::plan(future.callr::callr, workers=2)

before <- getFreeMemoryKB()
during <- run_furrr() 
after <- getFreeMemoryKB()
mem_message("callr terra         ", before, during, after)


future::plan("sequential") # reset plan
future::plan(future.callr::callr, workers=2)

before <- getFreeMemoryKB()
during <- run_placebo()
after <- getFreeMemoryKB()
mem_message("callr placebo       ", before, during, after)

Output: (Where during = mem free before - mem free during. after = mem free before - mem free after.)

sequential terra     mem use diff: -2 MB during, -9 MB after
sequential placebo   mem use diff: +9 MB during, -3 MB after
multisession terra   mem use diff: +459 MB during, +457 MB after
multisession placebo mem use diff: +13 MB during, +13 MB after
callr terra          mem use diff: +585 MB during, -5 MB after
callr placebo        mem use diff: +134 MB during, -8 MB after

[rhijmans]

Thank you for your elaborate example. I can reproduce it, but I do not immediately have an explanation. I will try to do some experimentation. For now, here is another way to estimate free memory that also works on OSX.

getFreeMemoryKB <- function() {
	x <- terra::rast()
       v <- x@ptr$mem_needs(terra:::spatOptions())
	v[2] / 128
}

I have now added that to terra as a new function called free_RAM, together with a function called mem_info . This does not address your question but these functions may be helpful for understanding this type of issue.

library(terra)
#terra version 1.2.11
mem_info(rast())

#------------------------
#Memory (GB) 
#------------------------
#available       : 46.48
#allowed (60%)   : 27.89
#needed (n=1)    : 0
#------------------------
#proc in memory  : TRUE
#nr chunks       : 1
#------------------------

free_RAM()
#[1] 48735816

[JimShady]

I've also seen this large memory behaviour when trying to use {terra} in parallel. I'll try and make a reprex at some point.

[HamishFathom]

I have also ran into this issue when using terra in a multiprocessing cluster. I am combining LiDAR into tiles and warping them to a standard resolution. From my anecdotal evidence it does not appear that too much RAM is being used per task, rather that the RAM is not being released at the end of each task, despite calls to rm() and gc(). The end result being that RAM use per instance steadily builds up until a crash occurs.

As a workaround I have tried splitting the tasks into smaller batches and looping over them (the RAM is returned when the cluster is closed). This works, but it does mean there is considerable dead time while longer tasks complete, and of course from the overhead involved in rebuilding the cluster.

I also tried forcing a restart of each instance when RAM use exceeds a certain threshold using free_RAM() and rs.Restart() but as far as I can tell this had no effect - I am not even sure if it is possible to restart a slave instance in this manner.

From memory I do not think this issue occurs in the raster package, so perhaps the best solution is to simply revert to using it for RAM heavy multiprocessing tasks. It is a sub optimal solution though, as I have found terra to be much faster.

For reference I am using the foreach package, and have tried both doSNOW and doParallel for cluster management with the same results. The OS is Linux. Version is terra 1.7.29

I would like to sign off by saying thanks for all the work on a great package.

[karldw]

@HamishFathom, you might be able to mitigate the issue by using something like callr to launch separate R sessions for each task. My example above uses future.callr, but you can also use callr directly. That way, you could at least avoid the overhead involved in rebuilding the cluster.

[HamishFathom]

@karldw, thanks for the tip, I have done a small test case callr and it does work.

However, I have managed to avoid the issue entirely by switching parallel package. Previously I was using the foreach package and was using either doSNOW or doParallel for cluster management.

I swapped to parallel and used parLapplyLB() and it has resolved the issue. So looks like I had a foreach problem rather than a terra problem.

[TylerLMcIntosh]

Has there been any progress made on figuring out what causes this behavior to occur? I have also been observing this behavior, but have an additional observation to add. (I am using the future & furrr packages for parallel processing)

I only have this problem when scaling up my parallelization on a multi-processing cluster or cloud computing cluster. When run locally, terra seems to properly control its RAM usage.