boulder icon indicating copy to clipboard operation
boulder copied to clipboard

FAKE_DNS doesn't work

Open liuqi-sun opened this issue 7 months ago • 2 comments

#And edit docker-compose.yml to change the FAKE_DNS environment variable to match. This will cause Boulder's stubbed-out DNS resolver (sd-test-srv) to respond to all A queries with the address in FAKE_DNS.

I modified the FAKE_DNS environment variable in docker-compose.yml according to the readme, set it to the ip of the host, and configured the domain name for the client to apply for the certificate in /etc/hosts of the host. However, boulder reported an error:no valid AAAA records found for example.com

docker-compose.yml : Image

boulder Image

liuqi-sun avatar Jun 16 '25 08:06 liuqi-sun

Is FAKE_DNS a DNS server?

liuqi-sun avatar Jun 16 '25 08:06 liuqi-sun

192.168.40.209 is the ip address of the host. I set FAKE_DNS to the ip address of the host and set host 192.168.40.209 example.com in the host /etc/hosts. My acme client is running on the host 192.168.40.209, and boulder is running in container mode on the host 192.168.40.209

liuqi-sun avatar Jun 16 '25 08:06 liuqi-sun

As you've figured out, FAKE_DNS is passed to chall-test-srv, which in turn is queried by the VA. So the VA should in theory connect to the IP address you put there when validating any hostname. Modifying /etc/hosts won't make a difference because the VA doesn't use the system DNS (except to find Consul).

I'm surprised you're getting "no valid A / AAAA records". I'm not sure why.

Incidentally, have you tried Pebble? https://github.com/letsencrypt/pebble/. What's your use case for Boulder?

jsha avatar Jun 16 '25 18:06 jsha

Do you have DNSAllowLoopbackAddresses set in your Boulder config? If not, the VA will not recognize reserved IP addresses (including in 192.168.0.0/16) in DNS results: https://github.com/letsencrypt/boulder/blob/c68e27ea6f10cd39d4053e1be7672758cc6217c9/bdns/dns.go#L414

A couple of notes about this config flag: it's important to turn it off in production, and we might potentially remove it from Boulder in the future.

jprenken avatar Jun 16 '25 18:06 jprenken

Do you have DNSAllowLoopbackAddresses set in your Boulder config? If not, the VA will not recognize reserved IP addresses (including in 192.168.0.0/16) in DNS results:

boulder/bdns/dns.go

Line 414 in c68e27e if ok && (policy.IsReservedIP(netIP) == nil || dnsClient.allowRestrictedAddresses) {

A couple of notes about this config flag: it's important to turn it off in production, and we might potentially remove it from Boulder in the future.

I haven't modified the Boulder config. After cloning the master code, I only modified the FAKE_DNS in docker-compose.yml

liuqi-sun avatar Jun 17 '25 01:06 liuqi-sun

As you've figured out, FAKE_DNS is passed to chall-test-srv, which in turn is queried by the VA. So the VA should in theory connect to the IP address you put there when validating any hostname. Modifying /etc/hosts won't make a difference because the VA doesn't use the system DNS (except to find Consul).

I'm surprised you're getting "no valid A / AAAA records". I'm not sure why.

Incidentally, have you tried Pebble? https://github.com/letsencrypt/pebble/. What's your use case for Boulder?

Are there any usage documents?Now I only found the readme and didn't find any other documents.Which is the configuration file for boulder and where is the API definition document of the configuration file?

liuqi-sun avatar Jun 17 '25 01:06 liuqi-sun

As you've figured out, FAKE_DNS is passed to chall-test-srv, which in turn is queried by the VA. So the VA should in theory connect to the IP address you put there when validating any hostname. Modifying /etc/hosts won't make a difference because the VA doesn't use the system DNS (except to find Consul).

I'm surprised you're getting "no valid A / AAAA records". I'm not sure why.

Incidentally, have you tried Pebble? https://github.com/letsencrypt/pebble/. What's your use case for Boulder?

Ok. Thank you. I'll try Pebble.

liuqi-sun avatar Jun 17 '25 01:06 liuqi-sun

same here... had to set it temporary back to release-2025-05-27.. hope it will be fixed soon 🤞

igolman avatar Jun 23 '25 19:06 igolman

As James mentioned, we recently changed a setting that is probably affecting you. We had a (slightly misnamed) config field called DNSAllowLoopbackAddresses, which allowed Boulder to interact with, not just 127.0.0.0/24, but an array of private-network IP addresses. Probably you are using a private-network IP address for your tests.

We recently reorganized our docker-compose.yml to use a non-private-network IP address for our tests, so we could simplify the logic a bit and hardcode the fact that Boulder should never connect to a private network IP address during validation. After that reorganization, we removed the DNSAllowLoopbackAddresses field from configs.

We later partially reverted that change, reinstating the config field (which is still used in some SRE dev processes), but continuing to omit it from out integration test configs.

It seems likely that you were broken by these changes, and manually editing "dnsAllowLoopbackAddresses": true into your boulder integration test configs might solve it for now. But note that we do not guarantee that will continue to work long term.

Out of curiosity, are you using this for an integration test environment with an ACME client, or something else?

jsha avatar Jun 23 '25 19:06 jsha

How to specify an external dns to use when resolving domain names? Is FAKE_DNS the ip address after domain name resolution or the address of the dns server used for domain name resolution? Now I want to specify an external dns server. The domain name returns the specified address, and the host at this address runs the acme client.

liuqi-sun avatar Jul 09 '25 08:07 liuqi-sun

As James mentioned, we recently changed a setting that is probably affecting you. We had a (slightly misnamed) config field called DNSAllowLoopbackAddresses, which allowed Boulder to interact with, not just 127.0.0.0/24, but an array of private-network IP addresses. Probably you are using a private-network IP address for your tests.

We recently reorganized our docker-compose.yml to use a non-private-network IP address for our tests, so we could simplify the logic a bit and hardcode the fact that Boulder should never connect to a private network IP address during validation. After that reorganization, we removed the DNSAllowLoopbackAddresses field from configs.

We later partially reverted that change, reinstating the config field (which is still used in some SRE dev processes), but continuing to omit it from out integration test configs.

It seems likely that you were broken by these changes, and manually editing "dnsAllowLoopbackAddresses": true into your boulder integration test configs might solve it for now. But note that we do not guarantee that will continue to work long term.

Out of curiosity, are you using this for an integration test environment with an ACME client, or something else?

As a test environment, I am implementing acme client

liuqi-sun avatar Jul 09 '25 08:07 liuqi-sun

FAKE_DNS is the ip address after domain name resolution. This works by passing it as a flag to chall-test-srv, and also by making challtestsrv the fallback resolver for the docker containers. A weird setup, I know! chall-test-srv sort of acts like an authoritative and sort of like a recursive (that never recurses).

$ rg FAKE_DNS -C 3
...
test/startservers.py
288-    # integration tests.
289-    challSrvProcess = run([
290-        './bin/chall-test-srv',
291:        '--defaultIPv4', os.environ.get("FAKE_DNS"),
292-        '-defaultIPv6', '',

If you want to point to a different recursive resolver, you'll want to modify the consul config for the relevant service (listed in va.json). Keep in mind Boulder only supports DoH resolution at this point! And will validate the DoH server's cert based on its local TLS trust settings, not the system ones. If you're trying to use an off-the-shelf public resolver it will probably not work well without a lot of fiddling. I'd recommend getting your test harness to work with chall-test-srv. Also please try Pebble if you haven't already: https://github.com/letsencrypt/pebble/

jsha avatar Jul 09 '25 18:07 jsha

FAKE_DNS is the ip address after domain name resolution. This works by passing it as a flag to chall-test-srv, and also by making challtestsrv the fallback resolver for the docker containers. A weird setup, I know! chall-test-srv sort of acts like an authoritative and sort of like a recursive (that never recurses).

$ rg FAKE_DNS -C 3
...
test/startservers.py
288-    # integration tests.
289-    challSrvProcess = run([
290-        './bin/chall-test-srv',
291:        '--defaultIPv4', os.environ.get("FAKE_DNS"),
292-        '-defaultIPv6', '',

If you want to point to a different recursive resolver, you'll want to modify the consul config for the relevant service (listed in va.json). Keep in mind Boulder only supports DoH resolution at this point! And will validate the DoH server's cert based on its local TLS trust settings, not the system ones. If you're trying to use an off-the-shelf public resolver it will probably not work well without a lot of fiddling. I'd recommend getting your test harness to work with chall-test-srv. Also please try Pebble if you haven't already: https://github.com/letsencrypt/pebble/ Thank you very much.

When I deleted FAKE_DNS and used an external dns server, I encountered the two problems you mentioned: 1.startservers.py reported an error, and boulder failed to start. I modified '--defaultIPv4', os.environ.get("FAKE_DNS") 2.Boulder only supports DoH resolution The configuration and logs are as follows:

Image Image I have a question. How can I set up bypassing Boulder to verify dns server certificates when using DoH

liuqi-sun avatar Jul 10 '25 02:07 liuqi-sun

I found that DOH can be disabled. After I disabled DOH, when I applied for a certificate using a specific dns server and observed the log, I found that the domain name was successfully resolved to the specified ip, and the acme client also sent a self-signed certificate. Why did it still report an error :During secondary validation: no valid A records found for www.liuqi.com; no valid AAAA records found for www.liuqi.com va.json is configured as follows: { "va": { **"dnsStaticResolvers": ["192.168.40.54:53"], "dnsAllowLoopbackAddresses": true,** "userAgent": "boulder", "debugAddr": ":8004", "dnsTries": 3, "dnsTimeout": "1s", "issuerDomain": "happy-hacker-ca.invalid", "tls": { "caCertfile": "test/certs/ipki/minica.pem", "certFile": "test/certs/ipki/va.boulder/cert.pem", "keyFile": "test/certs/ipki/va.boulder/key.pem" }, "grpc": { "maxConnectionAge": "30s", "services": { "va.VA": { "clientNames": [ "ra.boulder" ] }, "va.CAA": { "clientNames": [ "ra.boulder" ] }, "grpc.health.v1.Health": { "clientNames": [ "health-checker.boulder" ] } } }, **"features": { "DOH": false },**

liuqi-sun avatar Jul 10 '25 14:07 liuqi-sun

I found that DOH can be disabled. After I disabled DOH, when I applied for a certificate using a specific dns server and observed the log, I found that the domain name was successfully resolved to the specified ip, and the acme client also sent a self-signed certificate. Why did it still report an error :During secondary validation: no valid A records found for www.liuqi.com; no valid AAAA records found for www.liuqi.com va.json is configured as follows: { "va": { **"dnsStaticResolvers": ["192.168.40.54:53"], "dnsAllowLoopbackAddresses": true,** "userAgent": "boulder", "debugAddr": ":8004", "dnsTries": 3, "dnsTimeout": "1s", "issuerDomain": "happy-hacker-ca.invalid", "tls": { "caCertfile": "test/certs/ipki/minica.pem", "certFile": "test/certs/ipki/va.boulder/cert.pem", "keyFile": "test/certs/ipki/va.boulder/key.pem" }, "grpc": { "maxConnectionAge": "30s", "services": { "va.VA": { "clientNames": [ "ra.boulder" ] }, "va.CAA": { "clientNames": [ "ra.boulder" ] }, "grpc.health.v1.Health": { "clientNames": [ "health-checker.boulder" ] } } }, **"features": { "DOH": false },**

Image Image

liuqi-sun avatar Jul 10 '25 14:07 liuqi-sun

@liuqi-sun we've recently removed the DOH feature flag, so setting "DOH": false on a recent Boulder will have no effect.

If you're setting up integration tests and don't choose to rely on chall-test-srv for them, you will need to provide a suitable test environment for Boulder yourself, including a DNS resolver that offers DoH. Alternately, you'll find that Pebble is much easier to use for writing integration tests; that's still my recommended solution.

I'm going to close out this issue. I don't think we can provide further help on using Boulder for your integration tests, since it seems like your requirements (use an external DNS resolver) are not compatible with Boulder's test environment as currently implemented.

jsha avatar Jul 10 '25 17:07 jsha