thought-machine
Config and template cache URL(s) for remote_file rule to allow automatic fetching from a cache
The python_wheel rule allows a template CONFIG.PYTHON_WHEEL_NAME_SCHEME together with a CONFIG.PYTHON_WHEEL_REPO to define URLs which can be checked before downloading wheels from external libraries (i.e. PyPI).
The remote_file rule takes a list of urls to fetch from, which are checked in series until a fetch succeeds.
We would like to add similar config functionality as python_wheel to remote_file (which the wheel rule uses anyway under the hood).
We have devendorised most of our third_party dependencies. These now depend upon remote_file rules which download external resources as and when we plz build. The end goal is to plz build these targets and allow them to seamlessly fall back on a locally-managed storage bucket (e.g. S3 / cloud or local file cache) URL in case the external resource goes away / becomes unavailable. Just as happens for Python wheels with a custom storage bucket at present.
.plzconfig would provide something like:
[buildconfig]
cache-priority = low
cache-url = https://s3-zone.amazonaws.com/company-bucket
[remote_file]
name-scheme = {url_base}/{directory_name}/{file_name}
The priority above would indicate whether the cache took priority over the external resource.
This could be more generic and (perhaps eventually) could fully replace the python_wheel functionality; different rules would rely upon different name schemes and take different arguments to populate that string.
