tilequeue
tilequeue copied to clipboard
Rate limit WOF downloads
We're likely to hit rate-limiting issues from upstream, or random 503 errors, if we try to do large batches of operations. These should be fairly rare, as the file_hash
field in the WOF meta CSV should limit the changes, but sometimes it still happens that a lot of hashes change at the same time. We can probably spread out the impact and reduce the chance of failures by limiting each batch of downloads to a certain number of neighbourhoods each run.
Related: https://github.com/tilezen/tilequeue/issues/329