Upstream package sync replacement

[chefsalim]

trafficstars

The upstream 'sync' functionality that was enabled for the on-prem depot has a number of issues that need to be resolved. The issues range from operational (eg, setup/configuration/management - transparency of failures, resolving errors, etc) to user facing (eg, synchronous vs. asynchronous results, specific triggers that enable package sync, etc) to missing key functionality (eg, dependent packages or keys are not synchronized). The overall feature needs to be revisited and redesigned with a more detailed user story and feedback from the on-prem user community.

[HT154]

For my org, something similar to Artifactory's virtual repos would be ideal. Any upstream packages fetches would happen synchronously and transparently when first requested and cached for subsequent requests.

[jamessewell]

So in the meantime, is there a command which pulls a single package into a depot?

[HT154]

This worked for hab 0.59.0 (regressed recently, see https://github.com/habitat-sh/habitat/issues/5678)

# HAB_BLDR_URL and HAB_AUTH_TOKEN should be set in your env
function on_prem_sync() {
  PKG_IDENT=$1
  hab studio run "hab pkg install $PKG_IDENT --url https://bldr.habitat.sh -z '' && hab pkg upload /hab/cache/artifacts/$(printf $PKG_IDENT | sed 's/\//-/g')-x86_64-linux.hart"
}

Newer version have a force flag that would probably help too. This could also be worked into a real script that handles setting HAB_AUTH_TOKEN='' for the install command on newer versions.

[chefsalim]

@jamessewell Yes. the current plan is to have some helper commands (scripts) available to enable upstream pulls/updates until we get a better sync experience in place. Those should be available soon (before we pull the feature).

[predominant]

@chefsalim I'd love to see this return a 202 when a package is not yet available. The client would need to be modified to take that 202, add a delay and add to the end of the queue of things to pull down. This would allow the on-prem-builder time to fetch the requested package.

   +--------+            +-----------------+         +------------------+
   | Client |            | On|Prem Builder |         | Official Builder |
   +--------+            +-----------------+         +------------------+
       |                          |                           |
       |          get             |                           |
       |       core/foo           |                           |
       |+------------------------>|                           |
       |                          |                           |
       |          202             |                           |
       |    Hang on a minute      |                           |
       |<------------------------+|            get            |
       |                          |         core/foo          |
  +----|                          |+------------------------->|
  |    | Re-queue                 |                           |
  +--->|                          |<-------------------------+|
       |                          |
       |                          |
       |     Some time later      |
       |                          |
       |          get             |
       |       core/foo           |
       |+------------------------>|
       |                          |
       |      core/foo for you    |
       |<------------------------+|

[chefsalim]

Please see https://github.com/habitat-sh/on-prem-builder/pull/106 for a replacement script. This is not intended to be the 'ultimate' solution, rather a stopgap as discussed.

[chefsalim]

@predominant I'm not sure that is the model that we will ultimately land on. It's not clear that will be a great or transparent user experience to have a 202. Lets say the background sync attempt fails (upstream is unavailable then for example) - then should the client continue to keep waiting and for how long. How or whether to give the client status that something failed? Also there's not just one 'get' for a packages, we have many APIs like getting the latest metadata, search, etc. There's issues like that that get thorny in this model.

[predominant]

Understood

[jsirex]

There is also better, but harder method: you can issue a 302 redirect to a unique "task-page" which returns: "202" - while processing "5xx" - on error "200" - when ready

Download synchronously, like @HT154 said is also option.

I'm worried about real end-to-end use case. Let imagine we have:

1. on-prem https://bldr.local
2. secret/myproject habitat package (it depends on core/postgresql and on secret/libA)

I can try build the package with public builder -> no secret/libA -> no success. I can try build the package with private builder -> no package updates (like core/postgresql) So it is not clear for me how stuff can be automatically updated without explicitly upload them? Allow habitat to specify multiple repositories (like APT/YUM, MAVEN)? Make on-prem builder like supervisor - e.g. listen for updates in upstream builder and download them immediately?

If packages somehow get on-prem builder there will be no point to do "proxy download", "302-202-200 dance" - artifact either exists or not.

[HT154]

One major pain point that this issue would address is the constant missing packages after a core plans refresh. I usually spend a whole day or more dealing with this all over my build system:

↓ Downloading core/nghttp2/1.34.0/20190115160823
↓ Downloading core/nghttp2/1.34.0/20190115160823
↓ Downloading core/nghttp2/1.34.0/20190115160823
↓ Downloading core/nghttp2/1.34.0/20190115160823
↓ Downloading core/nghttp2/1.34.0/20190115160823
✗✗✗
✗✗✗ We tried 5 times but could not download core/nghttp2/1.34.0/20190115160823. Giving up.
✗✗✗

[jsirex]

I'm still in "developing-refactoring", but decided to push a scratch commit with hab-depot-sync. It, probably, buggy, but I was able to incremental sync only latest packages. Syncing doesn't require disk space.

Unfortunately you have to:

1. have ruby
2. clone https://github.com/habitat-plans/hab-sdk.git
3. rake build && rake install

Try hab-depot-sync --help

[jsirex]

Work still in progress. Now I have habitat:

sudo hab pkg install -b ya/hab-sdk -c unstable
hab-depot-sync --dest-depot https://bldr.on-prem.domain.local --dest-auth-token $auth_token --origin core --latest-version