grass icon indicating copy to clipboard operation
grass copied to clipboard

[Bug] r.patch fails with a large number of input maps

Open metzm opened this issue 1 month ago • 5 comments

Describe the bug

If a large number of input maps is provided as input to r.patch it fails with an OS error like "too many open files".

To reproduce

  1. Check the OS user limit of the number of open files with ulimit -n
  2. Create at least half the number of raster maps reprorted by ulimit -n
  3. Try to patch them with r.patch

Expected behavior

A large number of files can be patched.

Solution

Add the option file and the flag -z from r.series to r.patch

Additional context

Attempting to patch the results of AI models for semantic segmentation which require small input tiles of typically the size 256x256 pixels to 512x512 pixels.

metzm avatar Nov 14 '25 19:11 metzm

I am also experiencing this in a workflow on main. This wasn't happening ~ 5 months ago. This happens with 120 large tiles, a group of up to ~3 Gb. Machine should manage this without issue (64 Gb RAM, 4 Tb disk space). GRASS 8.4.1 does not have this problem.

dhdeangelis avatar Nov 20 '25 10:11 dhdeangelis

Add the option file and the flag -z from r.series to r.patch

Looks like these changes:

  • trac change
  • Clean-up of the trac change in https://github.com/OSGeo/grass/commit/b0a2ceecd207207e1456b143515fe060581a7517

neteler avatar Nov 20 '25 13:11 neteler

The reason why this fails with parallelization is that for each thread, each input raster is opened separately, i.e. each input raster is opened nprocs times, multiplying the number of open file descriptors. I am not sure how much sense this makes because reading is not parallel but serial?

metzm avatar Nov 20 '25 17:11 metzm

I am also experiencing this in a workflow on main. This wasn't happening ~ 5 months ago. This happens with 120 large tiles, a group of up to ~3 Gb. Machine should manage this without issue (64 Gb RAM, 4 Tb disk space). GRASS 8.4.1 does not have this problem.

The reason why r.patch recently (since about 5 months ago) fails with a large number of input maps is https://github.com/OSGeo/grass/pull/5731 merged on June 13 2025 which changed the default number of threads from 1 to the OpenMP default thread count.

metzm avatar Nov 20 '25 19:11 metzm

I am also experiencing this in a workflow on main. This wasn't happening ~ 5 months ago. This happens with 120 large tiles, a group of up to ~3 Gb. Machine should manage this without issue (64 Gb RAM, 4 Tb disk space). GRASS 8.4.1 does not have this problem.

The reason why r.patch recently (since about 5 months ago) fails with a large number of input maps is #5731 merged on June 13 2025 which changed the default number of threads from 1 to the OpenMP default thread count.

Thanks. I did not want to sidetrack this issue, so I posted a follow-up question in the discussions: https://github.com/OSGeo/grass/discussions/6641#discussion-9163330

dhdeangelis avatar Nov 21 '25 07:11 dhdeangelis