Support Build & Setup Environment Variables

[metaskills]

In my case I have a postCreateCommand that clones another organization repo. This works great under Codespaces but is hanging when using this project. I think what I see is that 🏃 start container will stop here.

Cloning into '/workspaces/mysharedrepo'...
Username for 'https://github.com/':

And never move from that spot. The lifecycle script is doing something like this.

git clone "https://${GITHUB_TOKEN}@github.com/myorg/mysharedrepo.git" /workspaces/mysharedrepo
cd /workspaces/mysharedrepo

Any recommendations to get that working?

[metaskills]

Would it make sense to have the env: variables be accessible to both the build and start stages? That might help here.

[metaskills]

FWIW, I would be happy if there was ANY environment variable during the build / start process so I could make some runtime decisions?

[Oblynx]

Hi! I wanted to request exactly the same feature.

I would like to run a postCreateCommand that pulls a private repo. I specify the secrets as environment variables, but they aren't picked up at the build stage. When I move the step that pulls the private repo to the runCmd, it works. Therefore, the only issue is that the environment variables specified in env aren't passed to the build phase.

[joshspicer]

Hey! This proposal makes sense to me. @stuartleeks I suspect you'll agree this is OK, but curious if there's any thoughts you have counter?

[stuartleeks]

I was wondering whether there would be any scenarios where we would want to set the build environment variables independently of the runtime ones.

E.g. would we want to have buildEnv and env, or just have env that is passed to both?

[Oblynx]

You can start with passing env to both. Then if an issue is reported with this behavior, we can split it up.

[metaskills]

You can start with passing env to both. Then if an issue is reported with this behavior, we can split it up.

Agreed here too. Thank you.

[Oblynx]

Hello! Just bumping this issue, is it being tracked/planned?

[stuartleeks]

I've was looking at adding support for this to the action, and was a little confused by the results of my testing for a while so thought I'd drop a comment here.

There's a PR to improve support for this (#211) but depending on how your devcontainer.json is set up, you may be able to make this work already.

To do this, you need to have the remoteEnv in your devcontainer.json referencing a value from localEnv:

	 "remoteEnv": {
		"YOUR_ENV" : "${localEnv:YOUR_ENV}"
	}

This instructs the devcontainer CLI to set the remote env in the container based on the localEnv context.

The localEnv context can be set using the env property at the step level (i.e. not nested under the with block):

      - name: Build and run Dev Container task
        uses: devcontainers/[email protected]
        env:
          YOUR_ENV_VAR: some_value
        with:
          runCmd: |
            make the-thing

#211 will add update the action so that env vars set in the env under the with block are passed to the container. This will provide a way to set container env vars without needing to have it set in devcontainer.json.

I'll raise an issue to add examples of this to the docs.

[metaskills]

I'm trying this out now and still can not figure out how to use an environment variable when the devcontainer is being built. I've read the docs in the PR work @stuartleeks did but still can't figure it out. Is the env only available to runCmd?

[stuartleeks]

@metaskills there is an example in the PR tests here

I added some initial extra content to the docs as well: https://github.com/devcontainers/ci/blob/main/docs/github-action.md#environment-variables

[moleary-gsa]

Hey Folks - I have been trying to get me head around this for the last 2 days with no luck.

I use a .env file for things like AWS credentials and database credentials.

When I run the following devcontainer.json locally it works fine:

// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/alpine
{
	"name": "Python",
	// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
	"image": "mcr.microsoft.com/devcontainers/python:0-3.10",
	// Set *default* container specific settings.json values on container create.
	"customizations": {
		"vscode": {
			"settings": {
				"python.languageServer": "Pylance",
				"python.linting.enabled": true,
				"python.formatting.provider": "black",
				"editor.formatOnSave": true
			},
			"extensions": [
				"ms-toolsai.jupyter",
				"ms-python.python",
				"ms-python.vscode-pylance",
				"njpwerner.autodocstring",
				"oderwat.indent-rainbow",
				"bungcip.better-toml",
				"ms-python.black-formatter",
				"donjayamanne.python-extension-pack",
				"donjayamanne.python-environment-manager"
			]
		}
	},
	// Features to add to the dev container. More info: https://containers.dev/features.
	// "features": {},
	// Use 'forwardPorts' to make a list of ports inside the container available locally.
	// "forwardPorts": [],
	// Use 'postCreateCommand' to run commands after the container is created.
	"postCreateCommand": "make setup",
	// Configure tool-specific properties.
	// "customizations": {},
	// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
	"remoteUser": "root",
	"runArgs": [
		"--env-file",
		"${localWorkspaceFolder}/.env"
	],
}

The critical part here is the postCreateCommand which is a make command that does a few things:

1. Download the AWS CLI
2. Install poetry package manager
3. Use AWS CLI to authenticate poetry to interact with our private Codeartifact repository
4. Install python dependencies (including one on our internal Codeartifact repository)

I know this should be prebuilt but I am taking things in stages as we incorporate dev containers into our standard way of working.

So when I rebuild this container locally, the runArgs entry pulls in a local .env file into the environment which works once it's there. The first time this repo gets cloned there is an error because the .env file is not available. I edit the repo in a recovery dev container, add the .env file with all the variables I need then rebuild. this works perfectly locally and only needs to happen once so it's not a huge deal.

When I try to run this in github actions there is literally no way I have been able to find to make this approach work and have spent almost 3 days fighting it now. Until I found this issue! I added the following section to the bottom of my devcontainer.json:

"remoteEnv": {
	"ENV": "${localEnv:ENV}",
	"AWS_DEFAULT_REGION": "${localEnv:AWS_DEFAULT_REGION}",
	"AWS_ACCESS_KEY_ID": "${localEnv:AWS_ACCESS_KEY_ID}",
	"AWS_SECRET_ACCESS_KEY": "${localEnv:AWS_SECRET_ACCESS_KEY}",
	"AWS_ACCOUNT_ID": "${localEnv:AWS_ACCOUNT_ID}",
	"CODEARTIFACT_DOMAIN": "${localEnv:CODEARTIFACT_DOMAIN}",
	"CODEARTIFACT_REPOSITORY": "${localEnv:CODEARTIFACT_REPOSITORY}",
	"CODEARTIFACT_USER": "${localEnv:CODEARTIFACT_USER}",
	"FC_USER": "${localEnv:FC_USER}",
	"FC_PASSWORD": "${localEnv:FC_PASSWORD}",
	"ENV_FILE": "${localWorkspaceFolder}/.env"
}

I add the following step to my job:

- name: Setup environment
        run: |
          echo "${{ secrets.ENV_FILE }}" >> $GITHUB_ENV
          echo "FC_PASSWORD=${{ secrets.FC_PASSWORD }}" >> $GITHUB_ENV
          echo "AWS_ACCESS_KEY_ID=${{ secrets.AWS_ACCESS_KEY_ID }}" >> $GITHUB_ENV
          echo "AWS_SECRET_ACCESS_KEY=${{ secrets.AWS_SECRET_ACCESS_KEY }}" >> $GITHUB_ENV

and this works fine in Github actions.

The issue is now when I build this locally, the variables specified in the remoteEnv object are not set so they don't get set in the container and the runArgs --env-file is completely ignored.

So I now can run my code locally or on GitHub actions, but not both!

Am I missing something ?

[moleary-gsa]

Indeed I was - the remoteEnv variables seem to be unsetting the variables that are set from .env if they are unset in the localEnv - is it possible to change that behaviour?

I got around this issue by the way; by setting a secret with the contents of the .env file we need in a CI context, and adding a step before the devcontainer is run which spits the secrets into a .env file:

...
- name: Setup environment
  run: |
    echo "${{ secrets.ENV_FILE }}"  >> .env
    cat .env
- name: Flake8
  uses: devcontainers/[email protected]
  with:
    cacheFrom: ${{ env.IMAGE_NAME }}
    push: never
    runCmd: |
      make flake8

The postCreateCommand runs before the runCmd is executed any time I use my pre-built container of course.

This also seemed to work by sending the secret to the github env $GITHUB_ENV variable so either could work I think

[stuartleeks]

@moleary-gsa - as you noted, using localEnv in the remoteEnv settings sets the value of the specified environment variable based on the local environment variable value. In the case of the action, this is the environment variable value that is present for that action (since that is what is passed to the devcontainer CLI running the steps). There's a section in the docs here

[moleary-gsa]

Thanks @stuartleeks - The issue I had was in trying to leave the .devcontainer.json alone i.e. not add any remoteEnv object to the file (because the .env file approach worked locally).

Anyway, It seems to work fine now.

I also wanted to highlight that the variables get completely unset when using remoteEnv if they are unset in localEnv. If this is intended it would be good to have that mentioned somewhere. I think the best solution (for me) would be to handle the case when a .env file is provided in addition to remoteEnv and the same variables are set in both places, having the variable that is not empty override the empty / unset one

[Oblynx]

@moleary-gsa what a nice idea! thanks!