tilequeue icon indicating copy to clipboard operation
tilequeue copied to clipboard

Rate limit WOF downloads

Open zerebubuth opened this issue 9 years ago • 1 comments

We're likely to hit rate-limiting issues from upstream, or random 503 errors, if we try to do large batches of operations. These should be fairly rare, as the file_hash field in the WOF meta CSV should limit the changes, but sometimes it still happens that a lot of hashes change at the same time. We can probably spread out the impact and reduce the chance of failures by limiting each batch of downloads to a certain number of neighbourhoods each run.

zerebubuth avatar Jan 11 '16 22:01 zerebubuth

Related: https://github.com/tilezen/tilequeue/issues/329

nvkelso avatar Mar 16 '18 20:03 nvkelso