juju
Handling included layers as dependencies (build from branch/commit)
We should provide charm authors with the option to build against branches (and/or commits) of the layers they are using.
Included layers are a dependency to the charm and we should have an official way to handle this kind of dependency in the same way we for example handle pip packages.
By building charms against branches of layers (not only from the master head) we enable charm authors to follow a more conventional workflow. I am pleased to see the recent set of opened issues that work towards exposing details of a build making it reproducible.
Layer authors can
- have stable branches,
- be able to do backwards incompatible changes,
- perform hot fixes,
- protect downstream charms from experimental/buggy code and features
The charm authors can
- protect themselves from unstable layers,
- have reproducible build artefacts,
- allow for charm bug fixes without bringing in new experimental/buggy code
I am well aware of the good practice we try to impose by not supporting builds from layer branches. We essentially force authors to fix broken layers and by doing so we promote sharing of layers and operational knowledge. However, the lack of branch support will probably force charm authors to fork all the layers they are using for the reasons presented above. We should consider is that by convention the master head is the most recent code. As such it tends to be unstable and has more bugs than a stable branch. And any charm you build is as stable as the most unstable layer used. I think this hurts adoption.
Finally I would like to bring to your attention the golang dependency management paradigm [1] that we could could easily follow.
Thank you for your consideration.
[1] https://golang.org/cmd/go/#hdr-Vendor_Directories
I definitely agree that we need to give more control to charm authors to ensure that they can reliably (re)build their charm, especially for patches and bug fixes, in the face of upstream layer changes. The main argument against vendoring or locking dependencies, be they layers, Python deps, or others, in the charm layer source is that it encourages "set and forget" behavior. I think we can have something that encourages updates while still allowing for charm author control, but we just have to be careful about the semantics and defaults.
The intended goal of #403, #404, #405, and #408 is to make it easy to say "build this charm as it was in charm store rev X, but with a specific set of updates" (whether those updates are in the charm layer, one or more base / interface layers, or in the wheelhouse). That is, to make it a build-time choice that has to be made every time, rather than something that is carried forward by default forever unless manually updated. The downside of that approach is that it requires more active maintenance on the charm author's part and doesn't allow for things like long-running "channels" for the layers (e.g., stable, beta, edge).
Perhaps a middle-ground would be to leave the layer specifications as-is but provide a charm command that creates or updates either a layers.lock file or vendored type cache. (Vendoring in dependencies is more guaranteed but increases the storage size for the charm layer, especially if we want to include Python deps by pre-building the wheelhouse. A lock file keeps things nice and small, but is subject to issues with the upstream repo and can't work offline.)
We should also think about the idea to support the notion of channels for layers, either via convention with branch names, or via the layer index (or some combination). I'm not sure how that would work with bzr or hg repos for layers, but if we add additional info to the index, I'm sure we can come up with something.
It's worth noting that the layers that are pulled down are already placed into the deps directory (which will be under $JUJU_REPOSITORY if set, or under the current directory), and you can use $LAYER_PATH and $INTERFACE_PATH to override layers from upstream. So technically speaking, it's already possible to vendor layers, but obviously that is a fairly clunky mechanism, as it stands. If we do decide to support a vendoring type solution for layers, we could build upon that.
The layers.lock file seems a very convenient approach. If I understand this correctly in the layers.lock you would be able to say
- "get layer L from the head of branch B", or
- "get layer L from latest tag"
- "get layer L from tag C.V.X with X being the maximum you can find"
- "get layer L from tag C.X.Y with X and Y being the maximum you can find"
- "get layer L from commit G"
During a charm build you would clone the layer L and point it to the desired branch/tag/commit. It is only option 5 that pins the code and it is something we do not want to promote. Given we have https://github.com/juju/charm-tools/issues/405 to recreate a past build I do not mind if we do not have option 5 at all.
I would downvote the idea of introducing the concept of channels since
- Tags and branches (marked as stable/unstable/featureX etc) are what people are used to express code maturity. I do not see the point of reinventing that.
- Channels alone are not enough, we should also have tracks that would map to backwards incompatible/future releases of a layer.
- It would add more complexity to our index.
It is true that we can support easily vendoring through $JUJU_REPOSITORY, $LAYER_PATH and $INTERFACE_PATH but we don't do so officialy. We could diconnect the dependency management steps from the build process. We could expect the user to update the dependencies via a layers.lock file right before she charm builds.
It doesn't address most of this issue, but it seems that layer authors can in fact currently specify branches in the repo URL for their layer in the layer-index, using the https://github.com/org/repo@branch-tag-or-sha convention.
I've been using git-subtree to vendor layers and interfaces. This allows pulling in changes from the upstream repository, and pushing them to a branch to propose changes back to the upstream.
I've got a few thoughts on this, as I've been facing this issue recently.
As Cory notes, there's concern a user could lock in to a specific tag and never get updates. I don't think that's entirely bad thing, though, if the tags follow a good practice.
Here's what I'm thinking.
There are latest tag which represents the most current stable release.
There are tags for each specific versioned release, i.e., v1.0, v1.0.1, v1.0.2, v2.0, v2.1, v2.1.1
There is a release tag for each major version, i.e., release-v1, release-v2, release-v3
As a user, I expect stability in my interfaces. If I subscribe to layer:mylayer@release-v1 I get whatever the last version of v1 is. Automatic updates for minor releases.
If I subscribe to layer:mylayer or layer:mylayer@latest, it pulls the latest table release, regardless of major version. Automatic updates for all releases.
