pmtiles upload is unreliable for large files

[bdon]

Look into S3 retries

[gimse]

Why is "pmtiles upload" needed? Is it not possible just to use the aws cli?

[bdon]

It isn't hard to implement as it's part of gocloud and it means one can make complete use of pmtiles with a S3-compatible bucket, a set of credentials, and a single CLI tool. But maybe the overlap of pmtiles users and those with access to awscli or rclone is 100%.

Last I checked the AWS CLI requires an installer step and multiple files which is more complex than a single-file binary like go-pmtiles. Thoughts?

[jamesscottbrown]

Personally I find pmtiles upload useful: the AWS CLI is a bit heavy to install, and rclone is more cumbersome to use. I thought rclone required first defining the S3 bucket that you want to use as a remote in a config file; IIUC, this is now longer true and you can now use the :backend:path/to/dir syntax, but it's still much harder to figure out the correct command for rclone than for pmtiles upload.

But maybe the overlap of pmtiles users and those with access to awscli or rclone is 100%.

I have access to both of these tools, but not necessarily from the machine or VM where I am generating the tiles. For example, I sometimes need to generate tiles containing data in a PostgreSQL DB that I can't connect to directly from my workstation because of firewall rules.

[lauriemann]

Just my .02, but I'd never used rclone until yesterday when I needed to copy up my .pmtiles files, but I found it pretty simple to use the docker version of rclone to upload to R2, so not a lot of value in spending time building yet another copy mechanism, time is better spent on other parts of the pmtiles project

[bdon]

Should be simplified and improved in 1.22.