pull: Support require-deltas=upgrade

[cgwalters]

Our general story for "production" repositories is that people use static deltas. This is part of a story of better supporting that. In this model we don't want clients to fall back to per-object fetch if due to some config/systems failure deltas don't exist.

This will also make it easier for us to potentially introduce a "deltas only" repository format in the future.

[cgwalters]

The recursive deltas issue is https://github.com/ostreedev/ostree/issues/470

As far as outdated clients, I'd say 2/3 of the burden here should fall on the OS vendor in providing a wealth of deltas. Really, disk space for this is so cheap in the world of S3/GCS etc.

Another thing we could do is encourage having --empty deltas (i.e. "from none") for the latest tip, so if your system is really outdated, you fall back to redownloading everything (i.e. the status quo for Docker/OCI), and for really outdated systems, RPM/deb type package systems start to approach redownloading everything anyways.

Another idea is some metadata in the summary file which instructs the client to only fall back to loose fetches if it's sufficiently old.

[miabbott]

Aplologies for not discussing the actual code here, but I wanted to throw in my thoughts on this approach.

That makes sense. But unless we start supporting recursive deltas, it's a bit hard to guarantee. E.g. what if a client hasn't been updated in a while and is behind by N commits, but the content provider only generates static deltas for the last N - 1 commits?

I share @jlebon's concern in this scenario. I don't think it would be much of a problem in a rigid, managed datacenter environment, but the community user that doesn't do upgrades frequently could get bit by this. (Related to this, on one of our internal mailing lists, I saw a message from someone who upgraded from F21 to F25 in one fell swoop. People like that user, while rare, would need a fallback.)

Another idea is some metadata in the summary file which instructs the client to only fall back to loose fetches if it's sufficiently old.

This idea sounds promising. Would like to hear more about how it could be accomplished.

[cgwalters]

Presumably this is a desktop system; anyone who was actually running Fedora 21 (or in general is not keeping firefox + kernel up to date at a minimum) is an irresponsible threat to the information security of their employer and their coworkers.

The fallback here is easy, edit the remote config to disable the flag.

Note also for Fedora today with multiple branches per major, it's a rebase to upgrade, which wouldn't fall under this category right now.

Anyways, big picture for the "really ancient" scenario, having the OS vendor do either "from none" deltas or just fall back to loose fetches is going to work OK.

Maybe though the flag should look something like require-deltas=upgrade=2 months or something - so the repo operator would provide deltas for the last 2 months of releases, and the client fall back if due to server misconfiguration/transient sync error the delta didn't exist.

[rh-atomic-bot]

:umbrella: The latest upstream changes (presumably c4f6522) made this pull request unmergeable. Please resolve the merge conflicts.

[cgwalters]

Labeling as needs-rework for design.

[openshift-ci-robot]

@cgwalters: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci[bot]]

@cgwalters: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/sanity	64c7d0c72d0d0c4bf058604cfd4fee18f5a81a9a	link	true	/test sanity

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.