devcontainers
Add support for alternate definition configs under .devcontainer
As part of the monorepo support for Codespaces, we want users to be able to choose a devcontainer specific for their scenario from a list of devcontainer definitions contained with the .devcontainer folder. For example, they could have .devcontainer/actions/devcontainer.json as well as .devcontainer/vscode/devcontainer.json and providing either path would work and successfully create the dev container specified at that path.
As an example, we'd expect for any of the follow paths to devcontainer definitions to be acceptable:
repository_root
| .devcontainer.json
|
| .devcontainer
|
+-- foo
| |
| +-- devcontainer.json
| \-- Dockerfile
|
+-- foo
| |
| +-- devcontainer.json
As a stretch, it would be great if the definition could be 2+ folders down such that .devcontainer/actions/allisb/devcontainer.json would also work but one subdirectory is sufficient for this first stage.
@chrmarti @bamurtaugh @joshspicer Yeah this fits pretty well with other requests we've seen to allow people to select in Remote - Containers. If detected, we prompt for which one to use there and it would be simple for unique configurations like you'd see in a monorepo.
This is also a separate and distinct problem from multi-container attach, which needs a different model. This can help with scenarios like https://github.com/microsoft/vscode-remote-release/issues/6188 too (via a common docker-compose.yml).
The idea of overrides / per-user config also could fit here given we've also talked about a possible parent property to allow for small variations in otherwise common config. One devcontainer.json could point to another in this same nested structure. That said, interestingly, using a local "feature" could allow you to share common content across the different devcontainer.json might be enough to get things going with just nested devcontainer.json files.
So riffing of the example above:
repository_root
| .devcontainer
|
+-- common-features
| |
| +-- features.json
| \-- install.sh
|
+-- foo1
| |
| +-- devcontainer.json
|
+-- foo2
| |
| +-- devcontainer.json
The devcontainer.json file could then contain:
"image": "foo1",
"features": { "../common-features#feature": "latest" }
In fact, this could help some definitions. We've seen PRs both that involve connecting to a device (like Flutter), and using a GPU (like Anaconda or CUDA) that can take advantage of a native device in some cases, but in others needs a different config to stay emulated or on the CPU. These definitions and repos can then have mostly the same config with a slight variation and a local path to a feature.
What both of you describe makes a lot of sense.
The devcontainer.json file could then contain:
"image": "foo1", "features": { "../common-features#feature": "latest" }
In the example above @chuxel, is the goal that either of the devcontainer.json's in foo1 or foo2 could contain this snippet, and users would want to create a set of features common to all /several of their definitions in the common-features folder?
@bamurtaugh Yep. You can do this with install scripts to some extent with a custom Dockerfile and a different context, but its clunky, and doesn't allow for extension install, etc. Features would resolve that.
Related: https://github.com/microsoft/vscode-remote-release/issues/1165, https://github.com/microsoft/vscode-remote-release/issues/2413
I wonder: Editors and IDEs tend to share project settings using a file or folder in the project folder (.vimrc, .sublime-project, .idea, .vscode). Aren't monorepos using one such settings file/folder per sub-project? Would it make sense to align with this and use one .devcontainer folder per sub-project in the monorepo? E.g., <root>/foo/.devcontainer/devcontainer.json and <root>/bar/.devcontainer/devcontainer.json for sub-projects foo and bar.
@chrmarti Would that look like any (or multiple) of the following?
Option one (supporting .devcontainer.json and .devcontainer in the root, just requiring .devcontainer for sub-projects. This would lead to nested .devcontainer folders):
repository_root
| .devcontainer.json
|
| .devcontainer
|
+-- foo
| |
| .devcontainer
| +-- devcontainer.json
| \-- Dockerfile
|
+-- foo
| |
| .devcontainer
| +-- devcontainer.json
Two (support .devcontainer in the root, but it wouldn't have nested projects / devcontainer folders and instead just a root-level devcontainer.json and Dockerfile):
repository_root
| .devcontainer.json
|
| .devcontainer
| +-- devcontainer.json
| \-- Dockerfile
|
+-- foo
| |
| .devcontainer
| +-- devcontainer.json
| \-- Dockerfile
|
+-- foo
| |
| .devcontainer
| +-- devcontainer.json
Three (don't use .devcontainer at the root, instead requiring each sub-project to provide .devcontainer.json, and still support a root .devcontainer.json):
repository_root
| .devcontainer.json
|
+-- foo
| |
| .devcontainer
| +-- devcontainer.json
| \-- Dockerfile
|
+-- foo
| |
| .devcontainer
| +-- devcontainer.json
Four (no dev container files/folders supported at the root):
repository_root
|
+-- foo
| |
| .devcontainer
| +-- devcontainer.json
| \-- Dockerfile
|
+-- foo
| |
| .devcontainer
| +-- devcontainer.json
I think my main questions are:
- Would we require a
.devcontainerfolder in each sub-project, and remove.devcontainerfrom the root? - Does this affect what the user can include in the root? i.e. still allow
.devcontainer.json, still allow or require.devcontainerwhich could include:- a root-level
devcontainer.jsonand Dockerfile, or - Nested
.devcontainerfolders
- a root-level
I wonder: Editors and IDEs tend to share project settings using a file or folder in the project folder (
.vimrc,.sublime-project,.idea,.vscode). Aren't monorepos using one such settings file/folder per sub-project? Would it make sense to align with this and use one.devcontainerfolder per sub-project in the monorepo? E.g.,<root>/foo/.devcontainer/devcontainer.jsonand<root>/bar/.devcontainer/devcontainer.jsonfor sub-projectsfooandbar.
@chrmarti @bamurtaugh Yes and no. Sometimes that's the case - but a monorepo can also be for a large single "application". Definitely heard feedback about role specific config in these cases - e.g. a front end developer verses back end, core services verses web app, etc. In these cases, it's the same abstract "project", but it involves shared libraries or services in the same repo, and different people working on it with different focuses. At scale, you end up needing multiple "views" on it - e.g. there's multiple .code-workspace files / solutions, etc on the same file tree.
In this case, each person's local machine is setup differently based on the role they play in developing the application. You want that same thing to happen when opening something in a dev container - its setup should be role specific for the same application. Right now, there's no good way to do this with devcontainer.json (without ugly hacks).
If the monorepo is the giant kind that actually crosses multiple applications, then yes, tool specific config would be under a sub-folder, but that's not what this is trying to solve. You can already have sub-folders with separate devcontainer.json folders - nothing in the spec prevents that.
The ask here is to deal with the scenario that is not well supported today that the Codespaces team has been hearing about frequently.
From my understanding, the main new features here is: VS Code searches for dev container definitions in root and sub-folders. Once VS Code detects all of the sub-definitions, it allows users to pick one when opening their project in a dev container.
Apologies if I'm missing this, but would this ask be just part of / one type of monorepo support? Here users will nest their sub-container definitions in one main .devcontainer folder at the root since this type of monorepo involves folks all on the same large repo with different roles. If users have dev containers spread into their sub-projects, rather than all in one central .devcontainer, that'd be what @chrmarti was suggesting and @Chuxel described as:
If the monorepo is the giant kind that actually crosses multiple applications, then yes, tool specific config would be under a sub-folder, but that's not what this is trying to solve
Apologies if I'm missing this, but would this ask be just part of / one type of monorepo support?
@bamurtaugh Yep, there's two types, one of which we already have covered in the spec even if just partly covered by the products.
-
Supported now by spec: Several independent "projects" in a single large repository. This is common with source control systems like Subversion, and in some cases orgs follow this practices in Git as well, but less commonly. Per the spec, you could run a "dev container CLI" in a sub-folder in the repository that itself has a .devcontainer folder in it. In this case, we still automatically mount the git repository into the container for image/Dockerfile, and users can control the mount in the Docker Compose case per the spec. So in concept, it works. (That said, Codespaces has product gaps here, and so does Remote - Containers for opening a repo in a container volume, but that's not a spec problem.)
-
Gap in spec: Several "views" on the same repository contents based on project/developer role or focus, rather than a single sub-folder. In this case, you likely need multiple sub-folders in the monorepo to build and develop the application. However, different roles have different focuses, extensions, settings, runtimes, etc. There's no single sub-folder to anchor to in determining what to include/not include. The request is to allow multiple variations under the root
.devcontainerfolder. Users that open the root folder can then make a choice as to which one they want to use, but still have access to the entire source tree as-needed. VS Code Workspace files and other tool's equivalents then allow you to filter down if desired. This particular style is pretty common in more recent cloud native/digital native application designs.
The "workspaceMount" property requires an absolute path for the "source", so you can't easily identify the source tree as being up a level, and even if you could its klunky. Of the two types, the Codespaces team is hearing more about the second case so far - which is why they raised the ask.
@chrmarti @bamurtaugh Given this has been picked up for implementation in what will become the reference implementation, I moved this one to "finalization." LMK if you disagree.
@Chuxel wouldnt it be better to simply share the vscode remote component or at last the installer so that people can simply deploy that and then set any location where that is deployed as target? this way all would be open as it would be needed.
@frank-dspeed Not quite sure what you are getting at by the "vscode remote component". Can you clarify? There's plans for a reference implementation (see #9) if that's what you're getting at, but that's not related to this topic.
What this is solving providing the ability to have multiple variations of a configuration in a central spot, while still retaining automount behaviors for the repository root. Anyone opening the root repository can them be presented with a list of existing configurations. You can place devcontainer.json contents in sub-folders today locally, though I know Codespaces if you are using that has not implemented that support.
@chuxel i mean the vscode server backend component that gets for example used when we add a remote ssh host then we could add containers also as remote ssh hosts or other protocols as needed and so work with any remote container.
as far as i am aware this is not in the vscode code base its a Microsoft Only thing. but when that would get open sourced and would get a api so that it is install able via a cli or something else then this would be the Solution that makes any one happy and even would open the abilitys to add total new scenarios.
it is a waist to keep it closed source and use it only for vscode ssh remote dev
see: https://github.com/microsoft/vscode/issues/89762 this makes all dev container features obsolet! It makes anything else that belongs to work with remote environments obsolet it is the sucessor of your iteration plan! And i know what i talk i was early adopter and contributor of the libcontainer project / docker / moby
also i am expert for the successor of containerisation that is WASM
but first things first promote that issue and you will see how fast the ecosystem growth
Thanks for the feedback, and very interesting - mainly just mentioning that it's a bit off topic for this issue.
What we are discussing here is really about the dev container specification rather than VS Code or the remote extensions themselves. The reference implementation will be OSS (#9) and cover how the needed metadata spins up related containers based on dev container config. This can be used anywhere.
However, since other non-VS Code tools can use it (e.g. CI systems, or anyone else that wants to support the spec), anything specific VS Code itself fits best with the issue you reference or one in https://github.com/microsoft/vscode-remote-release. VS Code injects what it needs based on the specific commit of the client, so there isn't a generic server pre-se. That said, you can use Remote - SSH and Remote - Containers in combination now if that is what you are getting at. (Docker does not need to be installed locally for that.)
@Chuxel thanks for the attention. Let me say something short about your offtopic concerns
Why is that related?
Your Dev Container overlaps with for example the Pod Definitions. Containers are already Nested Defined there are existing Opensource Definitions for that i do not want to list everything my conclusion that is related to the topic is that you should stop implementing an own Definition and should enable usage of Remote Containers as Dev Environment.
So the Missing part is always only the Interface to the Container or the Container Engine Not the Definition of Multiple Containers and the Operating System that they run or run not (Containers by design are like Unikernels so they should not depend on an OS at all its a Single Binary as WASM is) Containers are like a Compiler Target for LLVM
we got tools like podman (successor of docker) that got for example the so-called pod definition which includes everything needed. We can already define Dev Environment Pods the Project Eclipse Che does demo that well. we do not need a dev container spec as soon as we can configure our dev containers already. https://www.eclipse.org/che/ demonstrates that well with its Container Definition and Integration layer into Theia https://theia-ide.org/ Both got a Better Container Integration than this original vscode the only + point for vscode It got more extensions and more extension authors.
So Overall Vscode is the better Product but it leaks Remote Dev That demos well that the pod definition that Kubernetes and the docker successor podman are using is well defined already no dev container spec is needed on top of that only enable patterns are needed.
@frank-dspeed Happy to talk about this in another thread. I think that there's a bit of confusion here as to what this format is trying to do - which is not define container orchestration. I think what you are talking about is more relevant to #10 - since I think it's making the same point you are (if I'm understanding). There's no intent to replace pods or kubernetes with this format. In fact, the point of it is that this is metadata to work with existing orchestrator formats. I'd suggest we discuss there instead since we can confuse others thinking about this thread. The goal is to have structured data that can drop into multiple formats as the industry shifts over time - which is effectively the pattern you are describing. The single container model is there for simplicity to reduce concept counts for those that are not well versed in the containers ecosystem - but not to the exclusion of other things. A "dev container definition" (the contents of the .devcontainer folder) is intentionally multi-file for that very reason - you can stick a compose file there now, you could put another orchestrator format there later.
This particular issue is really about allowing you to pick from separate independent configurations that are still repository specific. In the case of a monorepo, you can have dozens of completely independent configurations and projects or just independant views, and this provides a convenient way to pick one from a set.
@Chuxel ok then we should talk here about terraform as that is covering exactly what you describe here https://www.terraform.io/ it describes containers and vms on diffrent infra.
Terraform is an open-source infrastructure as code software tool that provides a consistent CLI workflow to manage hundreds of cloud services. Terraform codifies cloud APIs into declarative configuration files.
I am not a big fan of re inventing the wheel and we got infrastructure as code call able via cli or api since over 20+ years now using json to describe infrastructure is simply the wrong direction.
Stuff like the JS Npm integration and the task runner is good enough for that. UI changes would help more then a new folder with a new own json formart that othert tools need to adopt again to reuse that codes.
we simply introduce all our old problems under a new umbrella. Thats what the community does since decades.
@frank-dspeed you are basically rooting for the deprecation of the .devcontainer specification in favour of terraform. This has nothing to do with this issue. But having the opportunity to specify and select from multiple devcontainer variations per project is imo a critical feature for larger projects that is currently missing.
@twsl nope terraform is only one of many implementations there are many more only wanted to point to one that is known do not get me wrong I am for innovation but I see innovation blocked because of definitions. There is a lot of overlapping functionality and it takes much time to educate developers. While the features are designed with the goal to less educate the developer. So I want to point out that I see a structural Root Problem with the definition and its progress of it.
The final goal is to enable the following use case:
- Git Clone Repo.
- Open in vscode
- Get an Offer that you can work in a defined dev container with a fully configured environment for the Project
only if that is wrong so when dev containers are designed for anything else. Then my suggestion is offtopic and then I am wrong. But if the above is true then I am fully correct with my observation I am 100% sure. But I also understand that I would need to write more Papers about that to convince people that do not have the same experience and knowledge. I Only point that out because i feel it is a urgent observation and the IDE is the new Teacher of Coders today. So if the teacher is wrong we get a whole bunch of Developers that get a wrong mindset implemented. Its like with Blockchain Containers are a buzz word for many people they get confused by the observations that they are doing they see them as VM's while they are by no means VM's they are Compiler Targets. They are hard linking dependencies.
Short Clarification about the replacement
The Replacement should be a better UI interface that allows to configure some vscode-remote host installs when they are on a container host then you can even integrate to run containers on that host and switch the ide into that for example when the remote-host-extension is installed you can even mount into local layers of the container daemon.
I already did Implementations for Web Agencys that are running for example Wordpress and Wordpress Plugin Development in that scenario we needed to integrate a interface into vscode that allows the developer to spin up wordpress installs, migrate data between installs and so on all on container host level else it would be time in efficent as it should speed up development.
All this and many more will never be possible with the current implementation and it will even block such multi container workspace scenarios that are well for developer groups that share a big container host (eleminates syncing problems and delays). as sayed i will need to write papers about that. But the only valid implementation i can think of can never be made with a .devcontainer folder storing files. We need install scripts and the ability to execute them remote then switch the IDE Into the Environment that we created. The Install scripts are the abstraction for the compiler settings so the compiler configuration what libs to install and link and how to link them into a final executeable image. we can execute multiple scripts in multiple containers running even on multiple CPU architectures. so This install scripts are already highly portable DevContainer Definitions.
the missing part is allow the IDE to Connect to the Remote host via diffrent protocols and take the filesystem and relay the shell commands and stdin and out.
this then can be expanded to some kind of remote process manager. (it is already like that because of the extensions host).
more easy switching between multiple vscode-remote-host installs is the key else the vscode-online or lets better say https://github.com/coder/code-server will be always a better fit for big teams as they can install that environment and switch dev spaces via the url also theia can do that.
but when vscode would be able to do that it would be the replacement for the dev container definition as the developer only needs the general aviable option to switch workspaces fast with same ide settings and the same main filesystem also useable so the remote and local one.
Please stop discussing this here - it is off topic for the specific issue. This sounds like the remote extension's attach functionality which is not related to this issue and will continue to exist. There's no pod or any other infrastructure definition in these configuration files. With the open sourcing of a CLI (#9), other infrastructure definitions can be referenced should communities want to contribute. What you describe is how things work already - the local filesystem does not have to be used and you can connect to a SSH host and work with containers from there, or you can just keep using attach. Both are fine. Other places to discuss could be #20 or #10 since I think things are already headed in the direction you describe.
Codespaces has gone ahead and implemented this capability (https://github.blog/2022-04-20-codespaces-multi-repository-monorepo-scenarios/). So, it's effectively part of the spec, but #9 will include it once released and Remote - Containers has not yet added support for making the selection via UX. I think we can close this out once #9 is released. @bamurtaugh @edgonmsft I know you folks are working on docs - I don't think this support is in the current PRs, correct?
Thanks for the tag, Chuck!
I don't believe so - is there a spot we're envisioning for it to be highlighted in spec materials? Perhaps in https://github.com/devcontainers/spec (@edgonmsft is working on other info for this repo)? I can add it to Remote-Containers docs too whenever it may be supported via the UI.
I just wanted to chime in. This feature is what I think I have been looking for. I work on many different open-source projects for my job and it would be of great value to define different image configurations to load easily. Behavior for tools I work on has different behaviors for various reasons that are subtle when running in let's say alma8 vs ubuntu, vs Fedora. It would be of great value for the community to be able to define a set of supported operating environments in the source project so anyone in the community working on that project can easily test the difference. The current limitation of only one environment is difficult to work with.
This is part of the spec, implemented in the reference implementation and supported by GitHub Codespaces and the Dev Containers extension for VS Code.