Get resource wait configuration

[mihail-ivanov]

• Add configuration for DOWNLOAD_TIMEOUT
• Add configuration and retries logic for get_resource in push_to_datastore job. If the uploaded resource is too big or the server have huge load, this call was failing which produces a lot of errors in the log.

[amercader]

get_resource just gets the resource metadata, not the actual file that resource['url'] points to, and the one that could be potentially big. So your improvements (which make sense) should apply to this request, unless I'm missing something?

[mihail-ivanov]

In our case the code was failing in both parts of the code. If the uploaded file was too big, get_resource needed more time to return the metadata, so we added more than one retries to do that. After that downloading the data for this file was taking more than 30 seconds, so we added a configuration option for DOWNLOAD_TIMEOUT.