datapusher-plus icon indicating copy to clipboard operation
datapusher-plus copied to clipboard

Smart resource download

Open jqnatividad opened this issue 2 years ago • 0 comments

Create a DOWNLOAD_ALWAYS_WHITELIST - a list of hosts from which DP+ will always download the entire dataset even if DOWNLOAD_PREVIEW_ONLY is true.

DOWNLOAD_ALWAYS_WHITELIST are typically local hosts or hosts where the CKAN instance has fast connections/peering arrangements.

Doing so will allow us to always do comprehensive metadata inferencing even if we're only pushing PREVIEW_ROWS into the Datastore.

jqnatividad avatar Feb 03 '23 13:02 jqnatividad