nx
nx copied to clipboard
nrwl
NX_* Environment Variables not working on prod builds in our test/staging deployments from gitlab, but only sometimes
Current Behavior
We use environment variables to point to different aws resources in our dev, test, and prod environments.
Locally, we use a .env file, and prefix all of the environment variables with NX_, like we are supposed to, and it all works fine.
In a Dev environment deployment that we set up that is built in a gitlab pipeline, and deployed to an S3 bucket, it also works fine with the env variables we set up in our gitlab CI/CD settings that are scoped to the dev environments.
However, in the exact same kind of pipeline for our Test (Staging) environment, it seems as if the NX build just simply won't recognize our environmental variables from our gitlab CI/CD settings that we have set there, but sometimes it does. We've tried adding the --skip-nx-cache to see if that was messing it up or not. Lately it seems like if we commit and push directly to our main branch, the build will include our env variables just fine, but if we make a merge request, and have a merge commit that results in the pipeline, none of our env variables get used and they are all undefined.
Here are our gitlab-ci steps for both of those pipelines. In the pipeline, we echo some of the variables that have been set as a sanity check to make sure gitlab itself is not misconfigured during the pipeline.
# store node_modules in a cache
cache:
untracked: true
key: '*******_portals_node_cache'
paths:
- node_modules/
# build apps dev
build_ap_dev:
stage: build
dependencies: []
environment:
name: ap_dev
action: prepare
only:
- develop
- /(^\d)*-feature-.*/
- /(^\d)*-bugfix-.*/
script:
- echo "cognito user pool - $NX_COGNITO_USER_POOLS_ID" # successfully echos here, so gitlab variable not a problem
- yarn build-ap --prod --skip-nx-cache
- echo "Build Successful"
artifacts:
expire_in: 4 hour
paths:
- dist/apps/agent-portal-frontend
build_ap_test:
stage: build
dependencies: []
environment:
name: ap_test
action: prepare
only:
- main
- /(^\d)*-hotfix-.*/
script:
- echo "cognito user pool - $NX_COGNITO_USER_POOLS_ID" # successfully echos here, so gitlab variable not a problem
- yarn build-ap --prod --skip-nx-cache
- echo "Build Successful"
artifacts:
expire_in: 4 hour
paths:
- dist/apps/agent-portal-frontend
package.json build scripts are kept simple, the pipeline adds the flags we need above
Other things from googling around, we've tried making sure our environment.prod.ts is like this:
export const environment = {
production: false,
};
Might this be related to some kind of Nx caching issue that gets stored in our node_modules cache for our pipelines?
Expected Behavior
Nx environment variables should work in all environments (dev, test, prod) when all of them have the same build command.
yarn build-appname --prod
Steps to Reproduce
- Set up environment variables on gitlab CI/CD that are scoped to specific environments
- Develop locally with a .env file in the root of the monorepo with variables that are prefixed with NX_*
- Make Merge Request to develop branch and merge - DEV environment build works fine.
- Make Merge Request from develop to main branch and merge - TEST environment build is missing env variables.
- Panic and directly commit some console.logs of env variables on the main branch and run a new pipeline in TEST env - It sometimes magically works, but sometimes not.
Failure Logs
When we download the artifacts from the build to look to see if the variables got used by Nx, we don't see them in a grep, is this normal?
Environment
env for my local machine, but problem happens in our pipelines that run on a node:15.14 docker image
nx report
> NX Report complete - copy this into the issue template
Node : 15.14.0
OS : linux x64
yarn : 1.22.5
nx : Not Found
@nrwl/angular : Not Found
@nrwl/cli : 12.6.4
@nrwl/cypress : 12.6.4
@nrwl/devkit : 12.6.4
@nrwl/eslint-plugin-nx : 12.6.4
@nrwl/express : Not Found
@nrwl/jest : 12.6.4
@nrwl/linter : 12.6.4
@nrwl/nest : Not Found
@nrwl/next : Not Found
@nrwl/node : Not Found
@nrwl/nx-cloud : Not Found
@nrwl/react : 12.6.4
@nrwl/schematics : Not Found
@nrwl/tao : 12.6.4
@nrwl/web : 12.6.4
@nrwl/workspace : 12.6.4
@nrwl/storybook : 12.7.2
@nrwl/gatsby : Not Found
typescript : 4.3.5
I noticed updating from nx 12.6.3 to 12.8.0, dotenv got updated. And looking at the releases, it seems to be in your interest to at least do this update.
update, we upgraded to the Nx 12.8 releases last week, and the first few deploys were successful. But now a few days later, with no package upgrades at all, we have had the exact same problem pop up again where we know that our gitlab env variables are being set correctly, but every other Nx build in our dev environment has undefined env vars
Hi, any solution so far? Our team is having the same problem on production builds with env variables being undefined
Hi, any solution so far? Our team is having the same problem on production builds with env variables being undefined
Yes, but it is a slow workaround. We added a line in our pipeline scripts to delete node_modules/.cache/nx so that is guaranteed to rerun the entire build step during the build step in our gitlab pipeline.
Hi, can this be prioritized please? Currently environment variables are not being read and it is a big issue
I'm not certain if it's an identical issue, but it sure seems similar: My project, running Nx v13.0.2 presently, has zero problems whatsoever with environment issues when started or built normally. However, when I run my Cypress e2e tests, they all disappear completely. This makes very little sense to me, since I assumed the same Webpack config was being used to build both versions out... but when I check my variables while running the watcher used to execute Cypress tests, all I see are these:
{
"NODE_ENV": "development",
"NX_CLI_SET": "true",
"NX_INVOKED_BY_RUNNER": "true",
"NX_WORKSPACE_ROOT": "<dir>/parfait-nx",
"NX_TASK_TARGET_PROJECT": "parfait-e2e",
"NX_TASK_HASH": "<hash>"
}
There are around a dozen environment variables (usually read from my .env file) missing from that readout.
@nurbashanghai - That depends. If you're having Kevin Vandy's problem, I can't help you. However, if you're having the same issue I was having (where the env vars don't populate for Cypress tests), you are in luck my friend!
I worked around this issue by manually bootstrapping the environment variables already available in my project, using the env-cmd NPM package. Here's the updated package.json command I use to launch my end-to-end tests:
env-cmd -f ./apps/myapp/.env nx e2e myapp-e2e
Eventually, I imagine the Nx team will find and fix this bug. The good news is that regardless of if/when/how they do that, this fix should remain working regardless.
We're having the same issue right now, using Azure pipelines. All environment variables set with a .env file are gone, it would be great if this gets addressed.
We have a very similar issue @KevinVandy, the env variables are cached between builds when going through a complete Git flow (push branch for PR review -> merge to dev -> PR dev to main -> merge to main).
If we push directly from the local main branch from the CLI to the remote main, the values are properly set.
This have a big impact for us because we control the meta robots index/follow through env variables, where we allow some content pages indexation only on production. At the moment, every time we push to main after going through a full Git flow, we have to do an extra & empty "magic" push directly from the main local branch (described above) to get the good values...
We are using Vercel for the deployments and we thought it was an issue on their end with the builds cache, but it might be more complex than that, based on several discussions I could find (including this one):
- https://github.com/nrwl/nx/issues/4662
- https://github.com/nrwl/nx/issues/4455
We are using Nx 12.7.2, we did not try to upgrade to the latest yet
Just FYI, upgrading to the latest NX 13.3.1 didn't fix the issue. It happens to us when using Dockerfile in the pipeline but it doesn't happen when running nx build from the pipeline itself.
At last, I found a solution by using dotenv-webpack plugin and passing a .env file to it! I tried it with env-cmd too but it didn't help while running nx build
@vsavkin ~ Any thoughts? It looks like env vars are broken for quite a few folks. Is this on the roadmap for a fix?
This is a major problem. I just recently started to use NX and now I'm wondering if I should back out. My prod builds are simply broken atm. It's kind of worst possible case.
This is a major problem. I just recently started to use NX and now I'm wondering if I should back out. My prod builds are simply broken atm. It's kind of worst possible case.
We have a solution to be clear, it just makes the builds go a lot slower.
Yes, but it is a slow workaround. We added a line in our pipeline scripts to delete node_modules/.cache/nx so that is guaranteed to rerun the entire build step during the build step in our gitlab pipeline.
This is a major problem. I just recently started to use NX and now I'm wondering if I should back out. My prod builds are simply broken atm. It's kind of worst possible case.
We have a solution to be clear, it just makes the builds go a lot slower.
Yes, but it is a slow workaround. We added a line in our pipeline scripts to delete node_modules/.cache/nx so that is guaranteed to rerun the entire build step during the build step in our gitlab pipeline.
That didn't work for me. Oddly my nestjs app picks up my environment from Kubernetes, but react doesn't. My issue may be different. But I'm not sure what to do either way.
Same problem here deploying to Vercel. I have multiple CRA & Nextjs apps in the workspace.
Can confirm that this is a problem for builds. I haven't tested it on Vercel, but running the built version of my app locally shows that the environment variables aren't being passed in.
I have not really had problems with environment variables, but found today that NX_INVOKED_BY_RUNNER doesn't seem to be set in some cases. I can't even find the code that sets it anymore. This breaks a whole bunch of our targets. Is this a known issue?
Have you all just investigate pure JavaScript files you have? As it's converted during compilation time.
I was able to get this working using runtimeCacheInputs something like:
"tasksRunnerOptions": {
"default": {
"runner": "@nrwl/workspace/tasks-runners/default",
"options": {
"cacheableOperations": [
"build",
"lint",
"test",
"e2e"
],
"runtimeCacheInputs": [
"echo $NX_ENV_VARIABLE",
"echo $NX_ENV_VARIABLE_1",
"echo $NX_ENV_VARIABLE_2",
"echo $NX_ENV_VARIABLE_3"
]
}
}
},
Maybe this might help someone else
I try to use Runtime Hash Inputs by watching some ENV changed and NX still not invalidate cache. But if I use
"targetDefaults": {
"build": {
"dependsOn": ["^build"],
"inputs": [
{
"runtime": "cat .env"
}
]
}
},
NX will invalidate cache, not sure this is proper way?
Any help with this? We still cannot seem to read the env vars on production.
I was having the same issue as well. My situation is that I'm using environment variables to determine if the environment is a preview or production. I would build out on staging and if the staging build is good to go, I'd rebase that onto production (which triggers a rebuild on Vercel). But, because of Nx caching, the env. vars would still be on staging, even though it was just rebuilt on production.
As soon as I implemented https://github.com/nrwl/nx/issues/6891#issuecomment-1110052748 solution, it now works as expected.
I was having the same issue as well. My situation is that I'm using environment variables to determine if the environment is a preview or production. I would build out on staging and if the staging build is good to go, I'd rebase that onto production (which triggers a rebuild on Vercel). But, because of Nx caching, the env. vars would still be on staging, even though it was just rebuilt on production.
As soon as I implemented #6891 (comment) solution, it now works as expected.
@l1qu1d Interesting, that did not work for me, in fact, the only thing on this issue that worked was https://github.com/nrwl/nx/issues/6891#issuecomment-1256641188. Thanks @tanint
I don't think this should be necessary, though nor am I sure what the proper solution is. It's very surprising that such an important issue is open for so long.
I was having the same issue as well. My situation is that I'm using environment variables to determine if the environment is a preview or production. I would build out on staging and if the staging build is good to go, I'd rebase that onto production (which triggers a rebuild on Vercel). But, because of Nx caching, the env. vars would still be on staging, even though it was just rebuilt on production. As soon as I implemented #6891 (comment) solution, it now works as expected.
@l1qu1d Interesting, that did not work for me, in fact, the only thing on this issue that worked was #6891 (comment). Thanks @tanint
I don't think this should be necessary, though nor am I sure what the proper solution is. It's very surprising that such an important issue is open for so long.
@benjaminhobbs I was stating my situation and what I believe is the cause, not a solution to the problem. The solution that you referenced is the same one that I did as well.