ISCE GPU

[10go-L]

Hello everyone,

I am currently facing an intermittent issue while using the GPU mode to execute TopoGPU and Geo2rdr in stackStripMap.py (specifically, run files 01 and 03). The problem arises inconsistently - sometimes the scripts work seamlessly, while at other times they fail.

Given that I am able to utilize the GPU for other tasks, I am inclined to believe that the issue is not related to the GPU itself but rather to how TopoGPU and Geo2rdr manage data processing.

Here's the error I encounter when executing Geo2rdr:

Copying Orbit and Poly1d data to compatible arrays...
Calculating relevant GPU parameters...
Available free gpu memory in bytes 140737383045008
and can process roughly 119107265 lines (each with 21100 pixels) per run.
NOTE: GPU will process image in 1 runs of 31243 lines


  ------------------ INITIALIZING GPU GEO2RDR ------------------

    Loading relevant geometry product data...
    Allocating memory...
    Done.
    Copying data to GPU...
    (NOTE: There will be 92 'empty' threads).
    Starting GPU Geo2rdr for run 0...
Sync kernel error: out of memory
Async kernel error: out of memory
    GPU finished run 0 in 0.112850 s.
    Copying memory back to host...
    GPU finished run 0 (with memory copies) in 0.776529 s.
    Cleaning device memory and returning to main Geo2rdr function...
  Writing run 0 out asynchronously to image files...
  Finished writing to files!

  ------------------ EXITING GPU GEO2RDR ------------------

Could anyone provide assistance in resolving this problem? Is it possible that this is simply a case of the GPU being unable to handle the volume of data required by these scripts?

Any help or guidance would be greatly appreciated.

Best, dll

[lijun99]

Can you try this temporary fix? Basically, it reduces the batch size to avoid the out of memory issue. Let me know whether it works so that I can decide whether to submit a PR.