grass-addons icon indicating copy to clipboard operation
grass-addons copied to clipboard

[Bug] r.accumulate fails with integer overflow in memory allocation

Open carolbarnes opened this issue 7 months ago • 5 comments

Describe the bug

r.accumulate extension consistently fails with an integer overflow in memory allocation, attempting to allocate approximately 18 exabytes regardless of input data size.

To reproduce

  1. Prepare a D8 direction raster (values 1-8) and a cost/weight raster
  2. Run r.accumulate with direction and weight parameters: gs.run_command("r.accumulate", direction="direction_raster", weight="cost_raster", accumulation="output")

Error occurs during "Allocating buffers..." phase

ERROR: G_realloc: unable to allocate 18446744047939747840 bytes of memory at /tmp/grass8-.../r.accumulate/grass_addons/src/raster/r.accumulate/accumulate_iterative.c:271

Expected behavior

r.accumulate should process cost-weighted flow accumulation with reasonable memory usage proportional to input data size (not in exabytes).

Screenshot

Image

System description

Operating System: Linux Rocky 9 GRASS GIS version: 8.4.1

Additional context

Always requests exactly 18,446,744,047,939,747,840 bytes Resolution independent: Same error occurs with 6.6M cells (30m resolution) and 202k cells (120m resolution) Format independent: Fails with both auto-detected and explicit format="45deg" Data independent: Occurs with valid D8 direction codes (1-8) and positive cost values

Input data characteristics:

Direction raster: Valid D8 format, integer type, ~34% null cells Weight raster: Positive cost values ranging 291-181,228, no obvious corruption Both rasters: Proper extents and data types

The consistent error size regardless of input suggests an integer overflow in buffer size calculation within the iterative accumulation algorithm.

carolbarnes avatar May 31 '25 09:05 carolbarnes

Thanks for reporting this issue. Please share your input rasters (direction and weights) as packs (r.pack) for my testing. Thanks!

HuidaeCho avatar May 31 '25 09:05 HuidaeCho

FYI, we have a separate repository for addons at https://github.com/OSGeo/grass-addons.

HuidaeCho avatar May 31 '25 09:05 HuidaeCho

FYI, we have a separate repository for addons at OSGeo/grass-addons.

(I have transferred this issue to the addons repo)

neteler avatar Jun 01 '25 08:06 neteler

Thanks! Input files as .pack created with r.pack are here https://github.com/carolbarnes/r.accumulationbug_inputs

Cumulative cost (weight) and movements direction (direction) were generated in r.walk.

carolbarnes avatar Jun 02 '25 09:06 carolbarnes

r.accumulate supports two flow direction formats: Image

Since your md_8dir is 1-8, it is recognized as the 45degree format (drainage in d.rast.arrow). Please see this one example of a circular flow pattern from md_8dir: Image

Flow accumulation is impossible in this encoding and this error occurs because it tries to accumulate flow indefinitely.

I assumed you used TauDEM to generate md_8dir so I converted your data to drainage (see how). I still got this circular pattern: Image

Based on these observations, either I don't understand your flow direction encoding (and r.accumulate doesn't support your format if there are no circular patterns) or you have sinks in md_8dir. In the latter case, you need to fill sinks first and calculate flow direction, or use r.watershed (no filling required).

In conclusion, this is not a bug in r.accumulate. It automatically detected the format of your data as drainage, but the data has circular patterns, causing infinite accumulation and a memory overflow. There is nothing to fix in r.accumulate and please check your data.

HuidaeCho avatar Sep 24 '25 11:09 HuidaeCho