stepman
stepman copied to clipboard
Fallback handling in case the StepLib (spec / source) is not available
Simply download the full spec (JSON) from s3 or from another Spec git repository, if the primary source can't be accessed (e.g. GitHub rate limiting)
Solution idea
- Include the "alternatives" / "caches" / "mirrors" list in the StepLib's git repo spec, in the steplib.yml
- The primary StepLib git repo still have to be downloaded at least once (e.g. from GitHub), but then the cache/mirror can be used in case the primary source (GitHub) is not available
- Cache / mirror should be either an URL (ZIP, tar) or a git (clone) url; a list should be allowed with priority - similar to how download_locations work for steps
- we should set up a mirror on GitLab for the main Bitrise StepLib - GitLab has an automatic GitHub mirroring feature
- maybe also set up an S3 cache, just to be sure & to test
- the ZIP/tar should not include the
.git
directory - to save space (.git
is pretty much the largest dir, as it includes diffs and git history)
Notes
Only download the full spec.json
, and do it only if there was a change. First send a HEAD request, to see when it was last modified, and store the modified date when downloading the spec.json
, so that it can be used for comparison in subsequent checks.