arcgis-python-api icon indicating copy to clipboard operation
arcgis-python-api copied to clipboard

copy_raster / _ImageryUploaderAGOL.upload_file fails for raster uploads around 50–60 MB

Open mirouml opened this issue 2 months ago • 2 comments

Describe the bug When publishing hosted imagery layers using copy_raster() (ArcGIS API for Python 2.4.2), raster uploads around 50–60 MB enter an infinite retry loop inside _ImageryUploaderAGOL.upload_file, never completing. Smaller and larger files upload successfully.

To Reproduce Steps to reproduce the behavior:

from arcgis.gis import GIS
from arcgis.raster import copy_raster

gis = GIS("home")
service_item = copy_raster(
    input_raster=r"C:\path\to\raster_51mb.tif",
    output_name="test_raster",
    gis=gis
)

error:

# Process stalls indefinitely repeating upload attempts
# No explicit exception is raised

Expected behavior Upload should complete successfully regardless of raster size.

Platform (please complete the following information):

  • OS: Windows 11 / AWS Lambda (AL2)
  • Python API Version: 2.4.2
  • azure-storage-blob: 12.17.x
  • Python: 3.11

Additional context Cause: upload_blob() performs a single PUT for < 64 MiB files; this often times out and loops indefinitely. Workaround: Replace the call with manual block upload:

import base64
from azure.storage.blob import BlobBlock
blob = self.container.get_blob_client(blobname)
block_ids = []
with open(file_name, "rb") as f:
    i = 0
    while chunk := f.read(4*1024*1024):
        block_id = base64.b64encode(f"{i:08d}".encode()).decode()
        blob.stage_block(block_id, data=chunk)
        block_ids.append(BlobBlock(block_id=block_id))
        i += 1
blob.commit_block_list(block_ids)

This resolves the issue for 50–60 MB rasters.

mirouml avatar Oct 30 '25 00:10 mirouml

@mirouml we have someone looking at this. Thank you for reporting it.

achapkowski avatar Oct 30 '25 10:10 achapkowski

@mirouml we have someone looking at this. Thank you for reporting it.

Thank you. I guess it might be worth to also consider to bump the azure-storage-blob (last week release 12.27.1). All the best and thanks again for looking into it.

mirouml avatar Nov 02 '25 23:11 mirouml