Medoid compositing method always running out of memory?

[orbitfold]

I'm interested in this approach but whenever I try it it always runs out of memory (even though other compositing methods work fine with the same settings)

[dugalh]

If the image computation is running over the user memory limit on download, you can reduce the max_tile_size parameter. I suggest trying max_tile_size=8 (MB). The command line docs are here and the API docs here.

[orbitfold]

Hi Dugal, 8 is what I have it set to after experimenting for a value that stops throwing those memory errors 99% of the time, I think something more is going on here. Does this method work for you?

[dugalh]

Hello, I have seen a "user memory limit exceeded" error when downloading a large Sentinel-2 medoid composite. When I decreased max_tile_size, it downloaded ok.

Earth engine has a size limit (32 MB) on download requests. geedim calculates the download tile sizes so that this is not exceeded. Earth engine also has a limit on user memory for image computations. This limit can still be exceeded when downloading computed images (e.g. medoid composites) with geedim. Unfortunately there is no way for geedim to know the memory requirements of a computation in advance. So it is up to the user to adjust max_tile_size if they run into memory issues.

Alternatively, exporting is not subject to the user memory limit, so a user could export their computed image to an asset, and then download the asset image with geedim.

If you think there is something else going wrong, please provide code to recreate the problem case.

[orbitfold]

I will try with even smaller tile sizes when I get a chance (next week)

[dugalh]

See the related documentation section in the latest release.

[orbitfold]

Well, for me it doesn't work even if I reduce max_tile_size to 0.1

[orbitfold]

Here is the GeoJSON I've used.

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {},
      "geometry": {
        "coordinates": [
          [
            [
              -0.3689373648039975,
              5.785841065493003
            ],
            [
              -0.3689373648039975,
              5.364440103464588
            ],
            [
              0.05576223573189054,
              5.364440103464588
            ],
            [
              0.05576223573189054,
              5.785841065493003
            ],
            [
              -0.3689373648039975,
              5.785841065493003
            ]
          ]
        ],
        "type": "Polygon"
      }
    }
  ]
}

And the code:

    polygon = data["features"][0]["geometry"]
    coll = gd.MaskedCollection.from_name(collection)
    coll = coll.search(start_date="2019-01-01", end_date="2020-01-01", region=polygon, cloudless_portion=cloudless_portion)
    comp_im = coll.composite(method=mosaic, region=polygon)
    comp_im.download(output_path, region=polygon, crs='EPSG:32735', scale=10, max_tile_size=max_tile_size)

Here mosaic = 'medoid', cloudless_portion = 0.0 and for max_tile_size I've tried values of 8, 4, 1 and 0.1

[orbitfold]

Btw, if it helps, the error seems to happen before it starts downloading

[dugalh]

OK. Can you tell me the value of collection?

[orbitfold]

'COPERNICUS/S2_SR_HARMONIZED'

[dugalh]

The latest v1.7.0 release contains some improvements and an example related to this.