stepman icon indicating copy to clipboard operation
stepman copied to clipboard

Fallback handling in case the StepLib (spec / source) is not available

Open viktorbenei opened this issue 8 years ago • 0 comments

Simply download the full spec (JSON) from s3 or from another Spec git repository, if the primary source can't be accessed (e.g. GitHub rate limiting)

Solution idea

  • Include the "alternatives" / "caches" / "mirrors" list in the StepLib's git repo spec, in the steplib.yml
  • The primary StepLib git repo still have to be downloaded at least once (e.g. from GitHub), but then the cache/mirror can be used in case the primary source (GitHub) is not available
  • Cache / mirror should be either an URL (ZIP, tar) or a git (clone) url; a list should be allowed with priority - similar to how download_locations work for steps
  • we should set up a mirror on GitLab for the main Bitrise StepLib - GitLab has an automatic GitHub mirroring feature
  • maybe also set up an S3 cache, just to be sure & to test
  • the ZIP/tar should not include the .git directory - to save space (.git is pretty much the largest dir, as it includes diffs and git history)

Notes

Only download the full spec.json, and do it only if there was a change. First send a HEAD request, to see when it was last modified, and store the modified date when downloading the spec.json, so that it can be used for comparison in subsequent checks.

viktorbenei avatar Apr 24 '16 10:04 viktorbenei