Cannot use local S3 storage

[jrounsley]

I am attempting to use optimise rasters against local S3 storage (not AWS). I can successfully create a profile against the storage and I have noticed the optimiseRasters python tool will not recognise the bucket/container parameter if I enter in an incorrect value. Therefore I am assuming that the tool is correctly looking up the local S3 bucket - similarly we hit errors if we purposefully enter in the wrong endpoint url or access keys.

What is the issue? When running optimise rasters tool we hit a GetBucketAcl error. The output messages confirm a logged warning that the machine could not connect to s3.amazonaws.com and the bucket key could not be found.

Can this tool be used against local S3 storage or is it limited to aws/azure/etc?

[Chamlika]

@jrounsley Optimizerasters has no local S3 support. Are you planning to run s3 clone locally to test custom lambda requirements combined with Optimizerasters as a client?

[jrounsley]

Our current and exploratory design have no requirement or use case for the lambda components of Optimizerasters.

We already have a design & capability that is working which uses optimizerasters (arcpy scripts) with the raw data output to a NAS file share. This process is kicked off by a virtual machine that leverages the arcgis pro python environment + the boto3 python dependencies.

We are exploring migrating this design to one where we run optimise rasters (same VM and python environment as above) and this outputs raster proxies and mrf_cache to fast flash storage with raw data output to local S3. This entire environment has no connection to the internet.

If optimise rasters cannot support local S3 then that is fine but it would be a nice enhancement for our (admittedly niche) use case.

[abhiataero]

@jrounsley can we connect, I would like to check few things, if it is s3 compatible local storage, then ideally optimize raster should work with it

[jrounsley]

Sorry you cannot connect.

As an example, I can successfully register the S3 bucket as a cloud store with ArcGIS Server and Pro. We can publish image services which use the S3 as the underlying storage. We cannot get OR to output to the S3 storage.

Let us know your thoughts and we can test more options/params within the profile editor and OR tool. The fact the error is reporting a failure to find the bucket key via s3.amazonaws.com is not a good sign.

[jrounsley]

Hi @Chamlika, just seeing if there is an update with this? It would be fantastic to utlise OptimizeRasters with our disconnected S3 storage.

[Chamlika]

Hi @Chamlika, just seeing if there is an update with this? It would be fantastic to utlise OptimizeRasters with our disconnected S3 storage.

Hi @jrounsley @abhiataero I'd hope everything is good with you! It's been a long pending requirement indeed. I'll get back into this issue to have an update to support S3 compatible local storage systems. To start off with, will use minio for compatibility tests and will have a response once we've something to release, thanks!

[Chamlika]

@jrounsley Please have a look with the latest release and let us know. Tested with local minio server to read and write locally. A sample profile is shown below. Also note, for now the addressing_style entry needs a manual update until the UI gets ready, thanks!

[minio] aws_access_key_id = AKIAIOSFODNN7EXAMPLE aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY aws_region = us-east-1 aws_endpoint_url=http://localhost:9000 addressing_style = path

[jrounsley]

Hi @Chamlika, thanks so much for getting back to me so quickly. It has taken a little bit of time to figure things out on my end to test the updated code, however I have run into an issue.

I have added the logfile OptimiseRasters-Logs.zip

Some files are getting added to my local S3, so the connection is working, however the TIFs are not copying up, and LERC files are not being generated.

Any suggestions on what I may be doing wrong?

Thanks again.

[Chamlika]

@jrounsley The log (XML) says the TIF files associated with the .TIL had failed. This may not be anything related to the local S3 connection. I'll have a test with a TIL sample on my side to see if anything related to TIL is broken and will get back to you, thanks!

[Chamlika]

@jrounsley I don't see any issue related to converting of TIL files on my side and could successfully upload to the local s3 Minio server. Do you think you can get me the TIL sample used with just one TIF file to keep the zip file size low together with the OR template used to help with the issue, thanks!

[jrounsley]

Hi @Chamlika thanks for your help - we got it working.