deno
deno copied to clipboard
denoland
Fetching https://1.1.1.1 or any other bare IP address fails with 'invalid dnsname'
HTTPS fetch with bare IPv4 address fails:
> deno run --allow-net https://deno.land/[email protected]/examples/curl.ts https://1.1.1.1/
error: Uncaught Http: error sending request for url (https://1.1.1.1/): error trying to connect: invalid dnsname
at Object.jsonOpAsync (core.js:236:13)
at async fetch (deno:op_crates/fetch/26_fetch.js:1272:29)
at async https://deno.land/[email protected]/examples/curl.ts:3:13
HTTPS fetch with bare IPv6 address fails:
> deno run --allow-net https://deno.land/[email protected]/examples/curl.ts https://[2606:4700:4700::1111]
error: Uncaught Http: error sending request for url (https://[2606:4700:4700::1111]/): error trying to connect: invalid dnsname
at Object.jsonOpAsync (core.js:236:13)
at async fetch (deno:op_crates/fetch/26_fetch.js:1272:29)
at async https://deno.land/[email protected]/examples/curl.ts:3:13
Real curl is fine:
> curl -sI https://[2606:4700:4700::1111] | head -n1
HTTP/1.1 200 OK
> curl -sI https://1.1.1.1/ | head -n1
HTTP/1.1 200 OK
HTTPS fetch with DNS name that resolves to said IP addresses works:
> deno run --allow-net https://deno.land/[email protected]/examples/curl.ts https://one.one.one.one | grep \\.\\.
............................................................
.........1............1............1............1...........
........11...........11...........11...........11...........
.......111..........111..........111..........111...........
......1111.........1111.........1111.........1111...........
........11...........11...........11...........11...........
........11...........11...........11...........11...........
........11...........11...........11...........11...........
........11...........11...........11...........11...........
........11...........11...........11...........11...........
........11...........11...........11...........11...........
........11...........11...........11...........11...........
........11...........11...........11...........11...........
........11...........11...........11...........11...........
........11....ooo....11....ooo....11....ooo....11...........
......111111..ooo..111111..ooo..111111..ooo..111111.........
............................................................
> deno --version
deno 1.4.1
v8 8.7.75
typescript 4.0.2
I'm trying to do something similar and I believe this issue has its roots deep down the Rust ecosystem.
Deno's fetch() is implemented in the op_crates/fetch crate which uses reqwest to perform the http requests.
Inside that Cargo.toml, we see that the reqwest build uses rustls (https://github.com/denoland/deno/blob/master/op_crates/fetch/Cargo.toml#L18)
There is currently a limitation using Rustls when sending http requests to plain ip addresses (see https://github.com/ctz/rustls/issues/281) which seems to stem from https://github.com/briansmith/webpki/issues/54, an issue which has been open for 3 years.
It seems there's not much you can do right now to get around this limitation apart from using a third-party request library which doesn't use rustls
https://github.com/http-rs/surf
This one doesn't rely on rustls, but it uses async-std instead of tokio
reqwest work perfect
Any plans to support this with the native fetch implementation? This would be really nice for home automation projects.
Edit: in the meantime, anyone who runs into this can create an entry in their hosts file as a workaround.
I think it's reasonable to look for alternatives to rustls that don't rely on webpki.
Even if the dns issue was resolved, there are other issues that seem to have just been abandoned, like support for self signed certificates.
I think it's reasonable to look for alternatives to rustls that don't rely on webpki.
rustls is an essential part of Deno as it is the entire backbone of all of our TLS operations. Replacing it would be a large refactor. The only viable alternative to rustls is OpenSSL, but using that has some downsides and comes with questions itself (eg. do we statically or dynamically link it?).
there are other issues that seem to have just been abandoned, like support for self signed certificates.
Deno supports self signed CA certificates through all available TLS APIs: --cert and DENO_CERT for all internal HTTP requests, caData option in Deno.createHttpClient for fetch, certFile option for Deno.connectTls, certFile option for Deno.listenTls. What exactly are you missing?
Also I want to note that Brian seems to have continued work on this in WebPKI, so maybe we will see some progress soon.
rustls is an essential part of Deno as it is the entire backbone of all of our TLS operations. Replacing it would be a large refactor. The only viable alternative to rustls is OpenSSL, but using that has some downsides and comes with questions itself (eg. do we statically or dynamically link it?).
Yeah there's no winning when it comes to TLS 😑😔
What exactly are you missing?
Don't get me wrong I think you guys did a great job! ❤️ ✨
My specific use case involves shipping a binary (thanks to deno compile! ✨ ), to end users that might have "whoknowswhat almost broken, but accepted by browsers" certificates on their machines, and expect them to work. 😕
Native IP addresses not working makes this already pretty difficult 😅.
This is complicated by rustls being such a stickler.
It wasn't really clear from the merge request I linked, but webpki flat out rejects self signed certs with the CA flag, with a CAUsedAsEndEntity error.
For my own certs that's fine, but I fear that there's a large-ish number of end users with self signed CA certificates that just copy pasted them from some tutorial. 😩
Anyways, looks like there's just waiting that Brian becomes more responsive, so let's hope for the best 😄😅.
It wasn't really clear from the merge request I linked, but webpki flat out rejects self signed certs with the CA flag, with a CAUsedAsEndEntity error. For my own certs that's fine, but I fear that there's a large-ish number of end users with self signed CA certificates that just copy pasted them from some tutorial.
This is untrue for Deno. We have tests validating that this works. You can use self signed CA certificates with Deno for all our TLS related APIs.
We have tests validating that this works.
Famous last words 😆
$deno test --cert "/Users/jp/.minio/certs/public.crt" --allow-net test/test_triblemq.js
Check file:///Users/jp/Desktop/triblesspace/tribles-deno/$deno$test.ts
running 1 tests
test Check loopback. ... Connected to ws://127.0.0.1:8816.
Sending fatal alert BadCertificate
Sending fatal alert BadCertificate
Sending fatal alert BadCertificate
[
TypeError: error sending request for url (https://localhost:9000/denotest/a15bb5cc5dea3b83a94e42d391a2309f1b4523ac07bad6f84b8ea681fd0e695e): error trying to connect: invalid certificate: CAUsedAsEndEntity
at processResponse (deno:core/core.js:223:11)
at Object.jsonOpAsync (deno:core/core.js:240:12)
at async fetch (deno:op_crates/fetch/26_fetch.js:1278:29)
at async S3Bucket.putObject (bucket.ts:445:18),
TypeError: error sending request for url (https://localhost:9000/denotest/9fd29d38aa625a97f70aa22f7dbec4eef1cf836587e28bc6f1eb236f8f06b241): error trying to connect: invalid certificate: CAUsedAsEndEntity
at processResponse (deno:core/core.js:223:11)
at Object.jsonOpAsync (deno:core/core.js:240:12)
at async fetch (deno:op_crates/fetch/26_fetch.js:1278:29)
at async S3Bucket.putObject (bucket.ts:445:18)
]
error: Uncaught (in promise) TypeError: error sending request for url (https://localhost:9000/denotest/2c22dc020c9fe79a0483e86dc1a23d6cf3a8f815e4802699599a8311f0142a2f): error trying to connect: invalid certificate: CAUsedAsEndEntity
at processResponse (deno:core/core.js:223:11)
at Object.jsonOpAsync (deno:core/core.js:240:12)
at async fetch (deno:op_crates/fetch/26_fetch.js:1278:29)
at async S3Bucket.putObject (bucket.ts:445:18)
Hello, this exact problem is probably off-topic for this thread, which is that Deno cannot connect to IP address hosts using HTTPS.
However, I can confirm that Deno has the above message when you try using a CA certificate directly on a server. I have a personal CA and just loaded up Deno HTTPS server using it. Resulting in:
> deno run --allow-net --cert /tmp/ca.crt https://deno.land/[email protected]/examples/curl.ts https://localhost:4443
Sending fatal alert BadCertificate
error: Uncaught (in promise) TypeError: error sending request for url (https://localhost:4443/): error trying to connect: invalid certificate: CAUsedAsEndEntity
at processResponse (deno:core/core.js:223:11)
at Object.jsonOpAsync (deno:core/core.js:240:12)
at async fetch (deno:op_crates/fetch/26_fetch.js:1278:29)
at async https://deno.land/[email protected]/examples/curl.ts:3:13
That being said, Google Chrome is also refusing to establish a connection:
This site can’t be reached The webpage at https://localhost:4443/ might be temporarily down or it may have moved permanently to a new web address.
ERR_SSL_KEY_USAGE_INCOMPATIBLE
as well as curl:
> curl --cacert /tmp/ca.crt https://localhost:4443
curl: (60) SSL certificate problem: unsupported certificate purpose
More details here: https://curl.se/docs/sslcerts.html
Wow, it looks like wget's only problem is the hostname being wrong 😅
> wget --ca-certificate /tmp/ca.crt https://localhost:4443
--2021-01-03 22:12:24-- https://localhost:4443/
Loaded CA certificate '/tmp/ca.crt'
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:4443... connected.
The certificate's owner does not match hostname ‘localhost’
Surely I'd be able to golf a bit with the certificate's usage flags to get one that works elsewhere, and while I'm there I could fix the CA flag as well.
It's unfortunate that the rust TLS ecosystem is acting a bit stagnant. I personally am quite blocked by the IP address issue, to the point where I'm writing code like Deno.run({cmd: ["curl", "https://....."]}). Hopefully some movement happens with the recent work on webpki's repo.
Any updates about the progress of a possible fix for this issue, it seems to have been slept on.
Any updates about the progress of a possible fix for this issue, it seems to have been slept on.
This has not been slept on. This is blocked on upstream missing support in rustls / webpki. If you want to accelerate this work I suggest you sponsor @ctz and @briansmith.
We have tests validating that this works.
Famous last words
@somethingelseentirely CA cert and TLS cert can not be the same - in you case it looks like they are.
Going to see if I can drive this forward a little bit, because.. well.. I love Deno, but this was rather painful for me.
So as I understand it, connecting to a Postgres Database with no DNS, just a hostname, using TLS, currently fails.
Small code sample:
await Deno.connectTls({
hostname: "34.76.80.151",
port: 5432,
certFile: "server-ca.pem",
});
Let's see if we can get that one step closer to working 💪 !
Straight off it's kinda odd that the docs give examples that don't work, and the default argument to Deno.connectTls will result in an error but rather than create PRs for that let's keep pushing in the other direction 😅 , making it work!
in the meantime, anyone who runs into this can create an entry in their hosts file as a workaround.
Based on this quote and my own use-case for this feature, perhaps a more practical step forward would be adding a parameter to replicate the servername option in Node.JS. The goal being that I could tell Deno what DNS name I expect the certificate to be good for. Then rustls should be able to validate successfully from there.
- In the GKE (Kubernetes) situation I can expect that a cluster IP address certificate is also always good for
kubernetes.default.svc. - In home automation perhaps the certificate is also good for
localhostor similar.
I'd be ok telling Deno those servernames ^^ if it means avoiding 'invalid dnsname'.
Of course the title issue should still be fixed, but upstream movement on it seems to be pretty slow.
If a DNS record has multiple entries can you force Deno in the fetch to try another entry?
If a DNS record has multiple entries can you force Deno in the fetch to try another entry?
I suppose the answer is no for the time being. We'll have to wait before Deno.CreateHttpClientOptions matures and gets a field to override the DNS resolution process for fetch calls.
I've been quiet for a year or so :) just checking in.
It looks like the error message is different nowadays (Deno 1.29.1) for IPv4: presented server name type wasn't supported
$ deno run --allow-net https://deno.land/[email protected]/examples/curl.ts https://1.1.1.1/
Sending fatal alert BadCertificate
error: Uncaught (in promise) TypeError: error sending request for url (https://1.1.1.1/): error trying to connect: presented server name type wasn't supported
Interestingly, IPv6 IPs still just report invalid dnsname 🤔
And, in fact, the request does now work with --unsafely-ignore-certificate-errors given to Deno, though I don't plan on using that flag in practice. Providing an alternate servername would be a safer workaround ^_^
Looking upstream, it seems rustls 0.21.0 will include a step towards this issue. I look forward to seeing where that goes:
- https://github.com/rustls/rustls/issues/184#issuecomment-1385458603
I forgot to mention a workaround: Deno.connect and Deno.startTls can be used together to provide different hostname values for the TCP and TLS aspects of a connection, instead of Deno.connectTls which uses the same hostname for both aspects.
Minimal example with https://1.1.1.1:
const tcp = await Deno.connect({ hostname: '1.1.1.1', port: 443 }); // the host we want to connect to
const tls = await Deno.startTls(tcp, { hostname: 'one.one.one.one' }); // the host we want to handshake with
// Send a raw HTTP request, and print the raw HTTP response:
await tls.write(new TextEncoder().encode('GET /healthz HTTP/1.1\r\nHost: 1.1.1.1\r\nConnection: close\r\n\r\n'));
for await (const text of tls.readable.pipeThrough(new TextDecoderStream())) {
console.log(text);
}
Also, I published /x/socket_fetch which implements a small subset of HTTP/1.1 in Typescript and can leverage Deno.startTls() as demonstrated above. Minimal example, again with https://1.1.1.1:
import { fetchUsing, TlsDialer } from "https://deno.land/x/[email protected]/mod.ts";
const dialer = new TlsDialer({ hostname: "one.one.one.one" });
const resp = await fetchUsing(dialer, "https://1.1.1.1/");
console.log(await resp.text());
This socket_fetch module is very limited (no POST, no socket reuse, etc) but works for some basic use-cases. I've been using it for HTTP GETs within Kubernetes clusters for the past year.
https://www.memorysafety.org/blog/rustls-new-features/
The first big feature is support for TLS certificates containing IP addresses. Rustls can now be used to set up TLS connections addressed by IP rather than a domain name. This is useful for things like Kubernetes pods, which often use IP addresses instead of domain names, and for DNS over HTTPS/TLS which need an IP address for the server to avoid circular dependency on name resolution. TLS certificates for IP addresses have been the most heavily requested feature for quite a while now and it's great to have it completed.
