features that depend upon features (composite features)

[johnnyreilly]

Heya!

I'm a keen user of dapr. I generally do this in the context of a devcontainer. I'm interested in how to migrate to doing the same using features. If I read things right, I need to use the docker-in-docker feature. On top of that I think I'd need to have some kind of dapr feature which would contain something like this:

RUN apt-get update \
    && apt-get install python3-pip -y \
    # Install Dapr
    && wget -q https://raw.githubusercontent.com/dapr/cli/master/install/install.sh -O - | /bin/bash \
    # Clean up
    && apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/* 

# Add daprd to the path for the VS Code Dapr extension.
ENV PATH="${PATH}:/home/${USERNAME}/.dapr/bin"

I'm guessing that there's some way to have "dependencies" for features? So the dapr feature would depend upon docker-in-docker. Am I headed in the right direction here? Or am I ... not?

It's possible that this is related: https://github.com/devcontainers/cli/issues/163

[joshspicer]

Hey, @johnnyreilly! With the current Features spec as it stands today, the default behavior is that the implementing tool decides the installation order. Our current goal with Features is that installation of one is atomic and idempotent (although we recognize that might not always be possible).

Currently we have two mechanisms to override the default installation order - they can be found in the "Installation Order" section of the spec.

I'm not too familiar with "dapr", but if "dapr" strictly requires the "docker-in-docker" Feature ahead of time, it sounds like the "installsAfter" property might be a good fit.

(We'd love your feedback either way on how to improve the Features spec)

[johnnyreilly]

Interesting. So for my own use case, I'd be looking to use docker-in-docker and building upon it with dapr. So I'd use "installsAfter" quite comfortably. But it's possible people might not be using docker-in-docker, in which case would this create a problem?

My guess is not , assuming installsAfter doesn't create a hard dependency on the mentioned features being installed. That assumption may not be valid though?

Where I am right now, I have quite a complex devcontainer setup, and I think features could massively simplify it

If you look at the devcontainer in this repo: https://github.com/johnnyreilly/dapr-devcontainer-debug-and-deploy

You'll see it's somewhat involved, with extra script files and whatnot. I think with features, that in the end that becomes much simpler. I just need:

• dotnet
• node
• docker in docker
• dapr

Would it help to flesh this out if I attempted to contribute a dapr feature to the repo? I could provide it with an installsAfter of docker-in-docker and see how it goes?

I do wonder if installsAfter should be opt-outable for people who are satisfying that dependency another way.

[Chuxel]

Yeah installsAfter affects ordering but does not create a direct dependency that forces the download. That allows you to alter behavior based on whether or not something has already happened. e.g., if you need python, you can say that your feature happens after the python feature, and if there's no python by the time your feature executes, then you can install the system one to meet the need. That avoids issues with multiple versions of something being installed and inflating an image size unexpectedly.

You can also ignore that ordering by specifying overrideFeatureInstallOrder in devcontainer.json as an opt-out to your point there.

However, there is the idea of a "composite" feature we could explore. In this case, your feature would have a hard dependency on another feature (likely a specific version of it as well). The thing we'd need to iron out is what happens when someone also specifies that dependency directly.

One possibility is it could behave like https://github.com/devcontainers/spec/issues/44, another would be that it literally fires twice, but then the dependent feature would need to be designed to handle that.

Let's move this discussion to the spec repo on the general topic of a composite feature.

[AlexanderLanin]

I was missing hard-dependencies since I started experimenting with features, but actually that would be a bad idea. The whole point of features is to avoid god like Dockerfiles. We want small reusable pieces. Dependencies between features make it impossible to use a different feature to install python. Or maybe it's already in my base image and I don't even need a feature at all.

My solution for the moment is to detect whether python is installed and in the correct version. If not, my feature exits with an error message telling the user to add python to the Dockerfile or to use *any* python feature.

Assuming features are named reasonably, what I would love is to have installsAfter: "*python*"

[joshspicer]

Assuming features are named reasonably, what I would love is to have installsAfter: "python"

Using a regex is an interesting idea. It would be assuming a lot, but I bet it would be right the majority of the time (and I suppose any hint is better than random?)

cc/ @chrmarti If not regexes, could be interesting to explore how else to drop general hints about install order.

[AlexanderLanin]

How about fallback to dependencies? Maybe that's some middle ground between hard dependencies and soft dependency order?

What I currently do:

if ! type python3 >/dev/null 2>&1; then
  echo "ERROR: Use a base image with python3 installed, add it to your Dockerfile or use a python feature like https://github.com/devcontainers/features/tree/main/src/python"
  exit 1
fi

How about a silent fallback, somehow supported by cli?

if ! type python3 >/dev/null 2>&1; then
  echo "INFO: python3 not installed, adding default python installation..."
  import ghcr.io/devcontainers/features/python:1
fi

[gund]

Hey, I just want to add one more use-case to the "features composition" proposal.

Basically I would like to be able to create a feature aggregate that will simply install other features, you can think of this as having a "preset" feature with maybe options to set versions and/or options of the other (sub)features it's going to install. This is basically what I would want to do to create my own custom dev environment in a single feature that I can take as a base and optionally add more features on top of it in different projects.

So maybe nesting infinitely many features inside features is not a very nice idea, so instead we could come up with the "preset features" that have a special ability to simply configure which features does it consists of and what it's default versions/options are, and then on the consuming side a user would be able to configure each sub-feature version/options as needed when using this "preset feature".

What do you think about this idea?

[joshspicer]

Hey @danielbraun89 - I've been taking a lot of inspiration from your excellent work on dependencies in devcontainers-contrib.

I'm curious if you have any learnings you'd like to share - I would be very interested to learn what you've learned through the process.

[danielbraun89]

Hey @danielbraun89 - I've been taking a lot of inspiration from your excellent work on dependencies in devcontainers-contrib.

I'm curious if you have any learnings you'd like to share - I would be very interested to learn what you've learned through the process.

Hoping for you the inspiration is not from a technical standpoint😛 as you must have seen my dependency implementation is the most primitive one you can think of. I think I did learn about a few challenges along the way tho, so for example:

Distinguish user intention

Lets assume a user has some kind of extended java feature. It wants the feature to be dependent on another feature, which he simply denotes with: ghcr.io/devcontainers/features/java:1

What was the user intention here?

Few options actually:

1. exended java feature simply requires a java JDK on the machine. Anything will do.
2. exended java feature requires a java JDK, of the "ms" distro, and of the "latest" version. (which is the default with ghcr.io/devcontainers/features/java:1)
3. exended java feature requires a java JDK, of the "ms" distro, and of the "latest" version, which was installed by the way of using SDKMAN! (which is again the default with ghcr.io/devcontainers/features/java:1) for example, because it further relies on sdkman for some reason.
4. exended java feature requires a java JDK, of the "ms" distro, and of the "latest" version, which was specifically installed using the specific ghcr.io/devcontainers/features/java:1 feature script, for example because it relies on any internal implementation detail that this feature script is known to have.

One solution for this might be asking for more explicit information from the user. Personally, my intuition is to use a sort of "condition" statement. So Lets say a feature author request a to be depend on

{ "feature" : "ghcr.io/devcontainers/features/java:1" , "options": {}, "condition": "type java" }

From the above it is expected that the java feature will be installed only where there is no java on the system all together In other words it convey intention 1

Another case:

{ "feature" : "ghcr.io/devcontainers/features/java:1" , "options": {"jdkDistro": "openjdk"}, "condition": "java -version | grep openjdk" }

So from the above, it is expected that the java feature will be installed only if a: there is no java on the system. b: there is but it is not of the openjdk distro. It is also expected that if the feature will be installed, it will install the openjdk distro specificly to achieve the needed outcome. In other words this convey (a variant) on intention 2

In similar ways you can convery 3 by checking the java installed location is on sdkman path, and 4 by using some metadata which will be have to be purposely be left there by the devcontainer (and this will have to be in the devcontainer spec in order for other implementors to act the same way)

Is it user-friendly? Admittedly not at all 😄 as I said this is just my go-to intuition and there must be a better way

A simplifying decision: Just assume one of the intentions above always holds true and stick to it. Yes this will leave leave some users in a position where there is no way to get their intention across, and the first issue the day after release would be "But how can I achieve the behavior of this and that". But maybe its unavoidable.

Catching circular dependencies:

This is a difficult one. Can you do it statically, on feature "compile time"? Probably yes for the current state, but obviously, because of the tolerance allowed in versioning, there is no guarantee that a later "1.0.X" version combination of the required features will not cause an endless loop. So this will probably have to be checked on runtime, but could be prior to actually running the installation process.

I'm not even talking about "diamond" dependency collisions (for both feature versions, and selected options). There is absolutely no way to solving those other than require people to pin their feature versions and create a version lock file, as many other full-fledge dependency managers do.

A simplifying decision: Simply do not allow specifying the version and the options to the feature dependency, the latest version and the default options will always be selected. This will probably limit the powerfulness of the dependency idea, and maybe will cause people to create multiple "variants" of some options as standalone features, in order to select a certain configuration. But again maybe its unavoidable...

Sorry it came little longer than I had intended (Me personally dont intend to invest any more time the on dependencies on my repo, and will simply use your implementation "as is", when it comes out)

[joshspicer]

An implementation based on https://github.com/devcontainers/spec/pull/234 has been merged and shipped with version 0.44.0 of the CLI. GitHub Codespaces and the Dev Containers extension will pick up the changes in the coming weeks (Dev Containers extension is released alongside the VS Code release cycle).

If you'd like to take advantage of these changes new changes, i'd suggest iterating on your Features directly with the dev container CLI, and staging the changes for when the functionality is available in the aforementioned tools!