App Service deploy with VNET integration http_proxy

[seansettle21]

Bug Report

• At what date and time did you most recently experience the problem? 5/27/2022 12:11am

• Where did you experience the problem? E.g. Azure Web Apps, Azure Functions, Azure Container Registry, or offline use. Zip Deploy to Azure App Services. App Service is VNET integrated with private DNS zones.

• If your repo is publicly available please share its URL: Not public, project is a nodejs project containing packages from an internal private repository.

• What happened? Oryx build fails in multiple ways when a proxy is required to access private network resources. If you have an app service and the {http,https}_proxy configuration settings exist, Oryx will fails to resolve the proxy

1. If no_proxy is not set or does not include "orxy-cdn.microsoft.io" Oryx fails at the "Detecting platforms..." step with "Error: Oops... An unexpected error has occurred." This isn't documented anywhere that I found except in other reported issues.
2. If no_proxy is correctly configured, the build then fails during the npm install step failing to resolve the proxy (ENOTFOUND), if an IP address is used instead of a private DNS name the error ECONNREFUSED

• What did you expect or want to happen? A successful deployment using VNET integration

• How can we reproduce it? In VNET integrated environment with a proxy set up to access private resources:

1. Set http_proxy and https_proxy configuration settings to point to the proxy by private dns name
2. In VSCode initiate a deployment from local folder
3. Observe deployment failure due to missing no_proxy setting during platform detection
4. Add no_proxy with oryx-cdn.microsoft.io
5. Attempt deployment again
6. Observe failure during npm install step with ENOTFOUND error of proxy name
7. Change proxy variables to use IP address
8. Attempt deployment again
9. Observe failure during npm install step with ECONNREFUSED

Note that in the Diagnostic Console or in a web SSH session to the running app service the proxy can be used successfully e.g. "curl -x http://proxy.svc.local:3128 www.github.com" works so VNET integration of the app service itself seems OK.

• Do you have log output? Please include between the backticks:

no_proxy not configured correctly

10:15:55 PM ecrs-poc-azapp-app-graphql01-test: Detecting platforms...
10:17:35 PM ecrs-poc-azapp-app-graphql01-test: Error: Oops... An unexpected error has occurred.
10:17:37 PM ecrs-poc-azapp-app-graphql01-test: /bin/bash -c "oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform nodejs --platform-version 16 -i /tmp/8da3f86d42053b7 -p compress_node_modules=tar-gz --log-file /tmp/build-debug.log  | tee /tmp/oryx-build.log ; exit $PIPESTATUS "
10:17:37 PM ecrs-poc-azapp-app-graphql01-test: Generating summary of Oryx build
10:17:37 PM ecrs-poc-azapp-app-graphql01-test: Parsing the build logs
10:17:37 PM ecrs-poc-azapp-app-graphql01-test: Found 0 issue(s)
10:17:37 PM ecrs-poc-azapp-app-graphql01-test: Build Summary :
10:17:37 PM ecrs-poc-azapp-app-graphql01-test: ===============
10:17:37 PM ecrs-poc-azapp-app-graphql01-test: Errors (0)
10:17:37 PM ecrs-poc-azapp-app-graphql01-test: Warnings (0)
10:17:38 PM ecrs-poc-azapp-app-graphql01-test: Deployment Failed.

proxy using name

12:37:49 AM ecrs-poc-azapp-app-graphql01-test2: Node Build Command Manifest file created.
12:37:49 AM ecrs-poc-azapp-app-graphql01-test2: Using Node version:
12:37:49 AM ecrs-poc-azapp-app-graphql01-test2: v16.14.2
12:37:49 AM ecrs-poc-azapp-app-graphql01-test2: Using Npm version:
12:37:50 AM ecrs-poc-azapp-app-graphql01-test2: 8.5.0
12:37:50 AM ecrs-poc-azapp-app-graphql01-test2: Running 'npm install --unsafe-perm'...
12:39:08 AM ecrs-poc-azapp-app-graphql01-test2: npm ERR! code ENOTFOUND
12:39:08 AM ecrs-poc-azapp-app-graphql01-test2: npm ERR! syscall getaddrinfo
12:39:08 AM ecrs-poc-azapp-app-graphql01-test2: npm ERR! errno ENOTFOUND
12:39:08 AM ecrs-poc-azapp-app-graphql01-test2: npm ERR! network request to http://dockercentral.it.att.com:8093/nexus/repository/npm-group/yocto-queue/-/yocto-queue-0.1.0.tgz failed, reason: getaddrinfo ENOTFOUND proxy.conexus.svc.local
12:39:08 AM ecrs-poc-azapp-app-graphql01-test2: npm ERR! network This is a problem related to network connectivity.
12:39:08 AM ecrs-poc-azapp-app-graphql01-test2: npm ERR! network In most cases you are behind a proxy or have bad network settings.
12:39:08 AM ecrs-poc-azapp-app-graphql01-test2: npm ERR! network 
12:39:08 AM ecrs-poc-azapp-app-graphql01-test2: npm ERR! network If you are behind a proxy, please make sure that the
12:39:08 AM ecrs-poc-azapp-app-graphql01-test2: npm ERR! network 'proxy' config is set properly.  See: 'npm help config'
12:39:08 AM ecrs-poc-azapp-app-graphql01-test2: npm ERR! A complete log of this run can be found in:
12:39:08 AM ecrs-poc-azapp-app-graphql01-test2: npm ERR!     /home/.npm/_logs/2022-05-27T04_37_50_404Z-debug-0.log
12:39:09 AM ecrs-poc-azapp-app-graphql01-test2: /bin/bash -c "oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform nodejs --platform-version 16 -i /tmp/8da3f9aa4f719f8 -p compress_node_modules=tar-gz --log-file /tmp/build-debug.log  | tee /tmp/oryx-build.log ; exit $PIPESTATUS "
12:39:09 AM ecrs-poc-azapp-app-graphql01-test2: Generating summary of Oryx build
12:39:09 AM ecrs-poc-azapp-app-graphql01-test2: Parsing the build logs
12:39:09 AM ecrs-poc-azapp-app-graphql01-test2: Found 0 issue(s)
12:39:09 AM ecrs-poc-azapp-app-graphql01-test2: Build Summary :
12:39:09 AM ecrs-poc-azapp-app-graphql01-test2: ===============
12:39:09 AM ecrs-poc-azapp-app-graphql01-test2: Errors (0)
12:39:09 AM ecrs-poc-azapp-app-graphql01-test2: Warnings (0)
12:39:10 AM ecrs-poc-azapp-app-graphql01-test2: Deployment Failed.
12:39:15 AM ecrs-poc-azapp-app-graphql01-test2: Deployment failed.

proxy using IP address

12:26:28 AM ecrs-poc-azapp-app-graphql01-test: Node Build Command Manifest file created.
12:26:28 AM ecrs-poc-azapp-app-graphql01-test: Using Node version:
12:26:28 AM ecrs-poc-azapp-app-graphql01-test: v16.14.2
12:26:28 AM ecrs-poc-azapp-app-graphql01-test: Using Npm version:
12:26:28 AM ecrs-poc-azapp-app-graphql01-test: 8.5.0
12:26:28 AM ecrs-poc-azapp-app-graphql01-test: Running 'npm install --unsafe-perm'...
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR! code ECONNREFUSED
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR! syscall connect
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR! errno ECONNREFUSED
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR! FetchError: request to http://dockercentral.example.com:8093/nexus/repository/npm-group/yocto-queue/-/yocto-queue-0.1.0.tgz failed, reason: connect ECONNREFUSED 10.87.2.16:3128
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!     at ClientRequest.<anonymous> (/tmp/oryx/platforms/nodejs/16.14.2/lib/node_modules/npm/node_modules/minipass-fetch/lib/index.js:110:14)
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!     at ClientRequest.emit (node:events:526:28)
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!     at onerror (/tmp/oryx/platforms/nodejs/16.14.2/lib/node_modules/npm/node_modules/agent-base/dist/src/index.js:117:21)
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!     at callbackError (/tmp/oryx/platforms/nodejs/16.14.2/lib/node_modules/npm/node_modules/agent-base/dist/src/index.js:136:17)
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!     at processTicksAndRejections (node:internal/process/task_queues:96:5)
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!  FetchError: request to http://dockercentral.example.com:8093/nexus/repository/npm-group/yocto-queue/-/yocto-queue-0.1.0.tgz failed, reason: connect ECONNREFUSED 10.87.2.16:3128
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!     at ClientRequest.<anonymous> (/tmp/oryx/platforms/nodejs/16.14.2/lib/node_modules/npm/node_modules/minipass-fetch/lib/index.js:110:14)
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!     at ClientRequest.emit (node:events:526:28)
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!     at onerror (/tmp/oryx/platforms/nodejs/16.14.2/lib/node_modules/npm/node_modules/agent-base/dist/src/index.js:117:21)
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!     at callbackError (/tmp/oryx/platforms/nodejs/16.14.2/lib/node_modules/npm/node_modules/agent-base/dist/src/index.js:136:17)
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!     at processTicksAndRejections (node:internal/process/task_queues:96:5) {
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!   code: 'ECONNREFUSED',
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!   errno: 'ECONNREFUSED',
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!   syscall: 'connect',
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!   address: '10.87.2.16',
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!   port: 3128,
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!   type: 'system'
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR! }
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR! 
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR! If you are behind a proxy, please make sure that the
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR! 'proxy' config is set properly.  See: 'npm help config'
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR! A complete log of this run can be found in:
12:27:45 AM ecrs-poc-azapp-app-graphql01-test: npm ERR!     /home/.npm/_logs/2022-05-27T04_26_28_994Z-debug-0.log
12:27:46 AM ecrs-poc-azapp-app-graphql01-test: /bin/bash -c "oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform nodejs --platform-version 16 -i /tmp/8da3f990e8b17e4 -p compress_node_modules=tar-gz --log-file /tmp/build-debug.log  | tee /tmp/oryx-build.log ; exit $PIPESTATUS "
12:27:46 AM ecrs-poc-azapp-app-graphql01-test: Generating summary of Oryx build
12:27:47 AM ecrs-poc-azapp-app-graphql01-test: Parsing the build logs
12:27:47 AM ecrs-poc-azapp-app-graphql01-test: Found 0 issue(s)
12:27:47 AM ecrs-poc-azapp-app-graphql01-test: Build Summary :
12:27:47 AM ecrs-poc-azapp-app-graphql01-test: ===============
12:27:47 AM ecrs-poc-azapp-app-graphql01-test: Errors (0)
12:27:47 AM ecrs-poc-azapp-app-graphql01-test: Warnings (0)
12:27:47 AM ecrs-poc-azapp-app-graphql01-test: Deployment Failed.
12:27:49 AM ecrs-poc-azapp-app-graphql01-test: Deployment failed.

[william-msft]

Hi @seansettle21, I noticed the proxy you mentioned is orxy-sdn.microsoft.io but the correct proxy is https://oryx-cdn.microsoft.io Would you be able to double check that? Also can you tell if the following environment variable is set? ENABLE_DYNAMIC_INSTALL=true

[seansettle21]

Sorry that was a typo, I fixed it in the description.

ENABLE_DYNAMIC_INSTALL=true is set if I log into the diagnostics console, but if I SSH to the app container it's not set.

[seansettle21]

Are there any configuration values I can verify or set to make the Oryx build work to use VNET integration? Currently I can only deploy to my App Service if I delete the package-lock.json file before I do the deployment in VS Code, which results in version drift between my dev environment and the app service instance.

[william-msft]

@vijay-microsoft @tulikac could there a be gap in our VNET integration with Oryx/AppService?

[giladk1221]

Hi, do you have any solution for this issue? when I added the relevant environment variables in the app settings it still dont install the build with Oryx.

[pauld-msft]

@giladk1221 Hello, can you please confirm for us that you have allowed traffic on port 443 to our oryx CDN (oryx-cdn.microsoft.io), as documented in https://github.com/microsoft/Oryx/blob/main/doc/hosts/appservice.md#network-dependencies? We improved logging around this issue so it is no longer generic, however it may not have rolled out to all of our partners.

[seansettle21]

@giladk1221 In my specific use case, I have a proxy server which I was able to route traffic through, as in that environment we are no allowed to have direct internet traffic out. Once I had the proxy admin add the oryx-cde.microsoft.io to the list of URLs allowed I was able to get this to work as expected by having the {http,https}_proxy environment variables set and a no_proxy which includes localhots and the IMDS meta endpoint.

[giladk1221]

I tried to add HTTPPROXY and NOPROXY configuration to app settings and open diagnostic shell(func app) via ssh and run a curl command to Oryx URL and it's opened via the proxy configuration. But when I publish a new version with the func command with the --remote-build flag its still notice that the URL for Oryx is not opened.

So it looks like that when we run the command for az funcapp publish --remote-build it's not using the proxy configuration that I defined in the app settings for the function app.

Do you know if I have any other option to add the proxy configuration in the function app environment variables? Or I must open my firewall rules behind the VNET to desired Oryx URL?

Waiting for your response

[simonjj]

We're closing this issue due to its age. If this is still impacting you please open a new issue and simply link back to this one.