feature: add support for custom/corporate certificate authorities

[clarkey]

We use an internal npm repository (Sonatype Nexus) which is has a https cert signed by an internal CA.

I have configured the registry correctly but still getting an error message which got me thinking it may be because it cannot verify the server. Could we have a way to load a cer string as an environment, or even a ca.cer file. Also a way to skip tls verification would be useful in general.

windmill-lsp-1 | Could not set npm package requirements. Error getting response at https://nexus-proxy.sdlc.redacted.net/repository/npm-group/ldaps for package "ldaps": An npm specifier not found in cache: "ldaps", --cached-only is specified.

--- DENO CODE EXECUTION ---

Download https://nexus-proxy.sdlc.redacted.net/repository/npm-group/ldaps
Sending fatal alert BadCertificate
error: Error getting response athttps://nexus-proxy.sdlc.redacted.net/repository/npm-group/ldaps for package "ldapts": error sending request for url (https://nexus-proxy.sdlc.redacted.net/repository/npm-group/ldaps): error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer: error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer: invalid peer certificate contents: invalid peer certificate: UnknownIssuer
    at file:///tmp/windmill/dt-worker-JPBHe-Y4s5i/01881224-74c7-b4bb-b1c6-d5521473e64c/main.ts:5:23

[rubenfiszel]

I wonder if it would be sufficient to add the certificates to the trusted store of the container.

[rubenfiszel]

@clarkey Were you able to make it work with the solution suggested above ?

[clarkey]

Hi @rubenfiszel

I tried adding corporate certificate as below.

FROM ghcr.io/windmill-labs/windmill-lsp:latest
COPY ca.crt /usr/local/share/ca-certificates/ca.crt
RUN chmod 644 /usr/local/share/ca-certificates/ca.crt
RUN update-ca-certificates

I verified this step to work by manually using wget inside the container to my NPM repository. However, it still did not resolve the issue. I am still getting this error message when running 'Test Script'.

ExecutionErr: ExitCode: 1, last log lines:
Download https://nexus-proxy.almuk.redacted.corp/repository/npm-group/ldapjs
Sending fatal alert BadCertificate
error: Error getting response at https://nexus-proxy.almuk.redacted.corp/repository/npm-group/ldapjs for package "ldapjs": error sending request for url (https://nexus-proxy.almuk.redacted.corp/repository/npm-group/ldapjs): error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer: error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer: invalid peer certificate contents: invalid peer certificate: UnknownIssuer
    at file:///tmp/windmill/dt-worker-ZZvsu-FhU7k/018921ca-26dc-89fc-7813-305c164cca16/main.ts:5:24

This is how I am using the new image within the docker-compose.yml, with the ca.crt installed and our own npm registry....

lsp:
    image: windmill-lsp-redacted:latest
    restart: unless-stopped
    environment:
      - NPM_CONFIG_REGISTRY=https://nexus-proxy.almuk.redacted.corp/repository/npm-group/
    expose:
      - 3001

I suspect Deno is not respecting the new ca.crt installed in the system store. Based on this https://github.com/denoland/deno/issues/5148#issuecomment-1376532117 I decided to try those two available methods for registering custom CAs inside Deno. Unfortunately setting both did not work either. Same error.

lsp:
    image: windmill-lsp-redacted:latest
    restart: unless-stopped
    environment:
      - DENO_TLS_CA_STORE=system
      - DENO_CERT=/usr/local/share/ca-certificates/ca.crt
      - NPM_CONFIG_REGISTRY=https://nexus-proxy.almuk.redacted.corp/repository/npm-group/
    expose:
      - 3001

Do you have anymore insight that I could use to try fix this issue?

[rubenfiszel]

Do the workers work correctly wrt to the ca-certificate on the other hand ? So that would be an issue specific to the lsp. If yes, then this might be caused by those env variables not being propagated correctly by the fork. I can investigate further.

[clarkey]

Ruben not sure what you mean in your first sentence there. Had anymore thoughts on this one?

[rubenfiszel]

Windmill has lsp for having an assistant in the webeditor and workers to actually run the job.

My question was around if workers were actually able to run the jobs and hence resolve your config registry. If yes, then it would be an issue isolate to the webeditor's smart assistants. Is that the case?

[clarkey]

The workers also throw an error which suggests a similar issue with env vars not being parsed correctly

ExecutionErr: ExitCode: 1, last log lines:
Download https://nexus-proxy.almuk.redacted.corp/repository/npm-group/ldapjs
Sending fatal alert BadCertificate
error: Error getting response at https://nexus-proxy.almuk.redacted.corp/repository/npm-group/ldapjs for package "ldapjs": error sending request for url (https://nexus-proxy.almuk.redacted.corp/repository/npm-group/ldapjs): error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer: error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer: invalid peer certificate contents: invalid peer certificate: UnknownIssuer
    at file:///tmp/windmill/dt-worker-ZZvsu-FhU7k/018921ca-26dc-89fc-7813-305c164cca16/main.ts:5:24

[rubenfiszel]

I verified this step to work by manually using wget inside the container to my NPM repository. However, it still did not resolve the issue. I am still getting this error message when running 'Test Script'.

And you did the same for the worker ? What about if you run a bare script by manually executing python x.py or deno run x.ts ?

[rubenfiszel]

We have some users reporting success on the worker by setting the certificates on /etc/ssl/certs. Could you reproduce ?

[SpiderD555]

Throwing my 3 cents here. I am also sitting behind company proxy with internal SSL certificate. When I put company CA certificate bundle into /etc/ssl/certs and update certificates and set environment variables (along with WHITELIST_ENVS as per your recommendation) and run a custom script in native worker docker container bash shell, then it works Here is the test script by the way:

export async function main(example_input: number = 3) {
// "3" is the default value of example_input, it can be overriden with code or using the UI
const res = await fetch(\`[https://jsonplaceholder.typicode.com/todos/3\`](https://jsonplaceholder.typicode.com/todos/3%5C%60), {
headers: { "Content-Type": "application/json" },
});
console.log(await res.text());
// return res.json();
}
main()

This is how I run it:

~# deno run x.ts                                                                                                                                                                                                                                          ✅ Granted net access to "jsonplaceholder.typicode.com".                                                                                                                                                                                                                    {                                                                                                                                                                                                                                                                             "userId": 1,                                                                                                                                                                                                                                                                "id": 3,                                                                                                                                                                                                                                                                    "title": "fugiat veniam minus",                                                                                                                                                                                                                                             "completed": false                                                                                                                                                                                                                                                        }

This makes me think there is a problem with proper variable propagation into the native worker or file access issue. Native worker somehow ignores custom certificate bundle altogether, not sure why is that.

[gbouv]

Hey @clarkey and @SpiderD555. We've made some changes in how handle custom certificates.

I think for the issues you are exposing above, using the DENO_CERT environment variable (or DENO_TLS_CA_STORE if the cert is trusted at the server level) should now work. Both for reaching private NPM registries and for native TS. Let me know if you still have issues.

In case you're interested, we published a deployment example in which we set up a docker compose with a private NPM registry (and Pypi) with custom certificates, and pull packages from it.

[SpiderD555]

@gbouv Just tried with latest docker image 1.224.1 (I build my own image with CA SSL certs included based on it), but without success. Native worker seems not to respect system SSL settings. I am using DENO_TLS_CA_STORE=system and passing it to execution environment using WHITELIST_ENVS=DENO_TLS_CA_STORE

Same script run on Deno non-native environment works just fine

Also tried the debug import import * as testpackage from "npm:@windmill/[email protected]" But I am getting certificate error: error: Error getting response at https://registry.npmjs.org/@windmill/helloworld for package "@windmill/helloworld": error sending request for url (https://registry.npmjs.org/@windmill/helloworld): error trying to connect: invalid peer certificate: UnknownIssuer: error trying to connect: invalid peer certificate: invalid peer certificate: UnknownIssuer

[clarkey]

Hey @clarkey and @SpiderD555. We've made some changes in how handle custom certificates.

I think for the issues you are exposing above, using the DENO_CERT environment variable (or DENO_TLS_CA_STORE if the cert is trusted at the server level) should now work. Both for reaching private NPM registries and for native TS. Let me know if you still have issues.

In case you're interested, we published a deployment example in which we set up a docker compose with a private NPM registry (and Pypi) with custom certificates, and pull packages from it.

I will try to check this week and get back to you. Thanks.