Added support for ddns lookups for addresses in access lists (resolved merge conflicts from #3364)

[sylphrena0]

Added support for ddns:somedomain.whateverddns.com address format in the client access lists.

This allows users to specify domain names instead of IP addresses for allow/deny lists, thereby allowing dynamic allow/deny lists.

This is useful if users have a service exposed to the public internet via a ddns domain, and they want to limit it so that only users from the local network can access the service on the ddns domain.

In theory, this is probably already possible by using custom DNS server to prevent any local network request to the domain name from going outside to the internet (and then setting allow list to local subnet in proxy manager), but not everyone can (or will) use a custom DNS server for their setup.

The new ddns support makes it trivially easy for anyone to limit to local network if they are using ddns without having to mess with custom DNS servers or network configuration. Also, if users want to expose their service to a fixed number of external users, then the ddns lookup can be used with the allow list provided the external users are using a ddns service. E.g. if I want to share a service on my network to 2 friends, and each friend uses a ddns that points to their public IP (friend1.domain.com, friend2.domain.com), then I can just add ddns:friend1.domain.com and ddns:friend2.domain.com to my allow list in proxy manager and they will continue to have access even if their public IP changes. I won't have to manually go an update the access list every time the IP changes.

This should address #1708 and #2240 .

Compared to some of the existing solutions mentioned in the above issues, this implementation should be the simplest with minimal overhead and no other dependencies (e.g. no cron, env vars, etc needed). Can directly specify the domains in the normal allow list UI.

Usage:

* Prefix the dynamic domain/host you want to use for the access list with `ddns:`
  e.g. If you want to add the dynamic hosts `yourdomain.ddns.com` and `yourdomain2.ddns.com` to your allow list, do the following:
  ![image](https://private-user-images.githubusercontent.com/651665/287489951-7c3327c8-ed4d-4a35-a768-7e8b96b63877.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDAyNjY5MjksIm5iZiI6MTc0MDI2NjYyOSwicGF0aCI6Ii82NTE2NjUvMjg3NDg5OTUxLTdjMzMyN2M4LWVkNGQtNGEzNS1hNzY4LTdlOGI5NmI2Mzg3Ny5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwMjIyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDIyMlQyMzIzNDlaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1iZjVjZGIxODA5ODllMTFkODk2M2VhNzAwMmU2M2YxYTg2ZDUzMTlkMTFhNjBjYTM4ZGI4OGFhMGUwMDk1NDljJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.aQFhRTrblpBwKkF901gYgL545SbkPZsYXBgctF_uwcA)

* Upon saving the access list, any associated hosts will automatically be updated to use the resolved IP of the dynamic domain/host in the access list (and this will trigger a nginx reload so that changes take effect).

* The proxy manager will also periodically poll the dynamic domains, and update any proxy hosts that are using those domains if there is an IP address update. Default interval is 1 hour, can be configured by setting the `DDNS_UPDATE_INTERVAL` env var to the desired number of seconds (minimum 60).
  On start up, all used domains will be resolved and any associated hosts will be updated in nginx about 10s after the proxy manager starts (10s buffer ensures server has enough time to finish loading).

Disclaimers:

* I haven't used js in almost 6 years, do not be surprised if the code is inefficient / not following best practices. Suggestions for cleaning up and refactoring the code to make it more efficient/readable are welcome!

* I'm using `getent hosts <hostname>` to look up the IP of the user defined domains - if there is a better way, please let me know and I can update the PR. I've tried to make it safe by using spawn instead of exec to prevent issues with unsanitized user inputs, however I'm not doing any custom sanitization.

Resolves conflicts from @vari's #3364 with a merge commit.

[1A3Dev]

I've just given this a try and it doesn't seem to work for me? I tried adding a dynamic domain to one of my existing access lists prefixed with ddns: however looking at the logs it didn't seem to recognise it as a DDNS IP?

[DDNS     ] › ℹ  info      DDNS Update Timer initialized (interval: 3600s)
[DDNS     ] › ℹ  info      Checking for DDNS updates...
[DDNS     ] › ℹ  info      Found 0 address(es) in use.
[DDNS     ] › ℹ  info      0 DDNS IP(s) updated.
[DDNS     ] › ℹ  info      Finished checking for DDNS updates

[sylphrena0]

I've just given this a try and it doesn't seem to work for me? I tried adding a dynamic domain to one of my existing access lists prefixed with ddns: however looking at the logs it didn't seem to recognise it as a DDNS IP?

[DDNS     ] › ℹ  info      DDNS Update Timer initialized (interval: 3600s)
[DDNS     ] › ℹ  info      Checking for DDNS updates...
[DDNS     ] › ℹ  info      Found 0 address(es) in use.
[DDNS     ] › ℹ  info      0 DDNS IP(s) updated.
[DDNS     ] › ℹ  info      Finished checking for DDNS updates

Yep, I ran into the same issue. I'm guessing there have been other changes since the previous PR, I'm going to actually dig into the code instead of doing a simple merge and hoping the previous implementation holds up.

[1A3Dev]

Yep, I ran into the same issue. I'm guessing there have been other changes since the previous PR, I'm going to actually dig into the code instead of doing a simple merge and hoping the previous implementation holds up.

I've just done a tiny bit of testing and discovered that it recognises the DDNS IP(s) if I restart the proxy after adding one. However there's also another issue, the IP that is being resolved for me is IPv6 not IPv4 which results in me still not having access 😂

[sylphrena0]

I've just done a tiny bit of testing and discovered that it recognises the DDNS IP(s) if I restart the proxy after adding one. However there's also another issue, the IP that is being resolved for me is IPv6 not IPv4 which results in me still not having access 😂

Yeah, I'm also not sure why they used the ddns prefix instead of RegEx to match valid domains/subdomains, since no other access hosts would accept a domain. I'm looking into it, but do let me know if you find any solutions!

[sylphrena0]

I pushed a change that should resolve addresses to ipv4 and changed the syntax of ddns entries to remove that prefix. I'm still investigating why entries only work after a restart

[1A3Dev]

If the IP fails to resolve and fallbacks to the domain, it currently still gets added to the nginx config which causes

nginx: [emerg] invalid parameter "subdomain.domain.com" in /data/nginx/proxy_host/1.conf:80

which initially is given as a warning in the logs (the website just shows a popup saying "Internal Error") however if you then restart the proxy, nginx fails to start

[sylphrena0]

If the IP fails to resolve and fallbacks to the domain, it currently still gets added to the nginx config which causes

nginx: [emerg] invalid parameter "subdomain.domain.com" in /data/nginx/proxy_host/1.conf:80

which initially is given as a warning in the logs (the website just shows a popup saying "Internal Error") however if you then restart the proxy, nginx fails to start

Good catch. The previous author added a check for this in the ddns_updater.js, but not to nginx.js. I implemented the same check in nginx.js and updated the logic to remove any existing value if it fails. I'm thinking that would be the correct choice, in case a ddns record was purposefully deleted, access will be removed.

[nginxproxymanagerci[bot]]

Docker Image for build 16 is available on DockerHub as nginxproxymanager/nginx-proxy-manager-dev:pr-4386

Note: ensure you backup your NPM instance before testing this image! Especially if there are database changes Note: this is a different docker image namespace than the official image

[EricFROL]

Yea this works as intended, but when applying changes in ACL tab I get a "400 internal error" that shows no message, but actually after hitting F5 the changes are saved and ends up working.

[1A3Dev]

when applying changes in ACL tab I get a "400 internal error" that shows no message, but actually after hitting F5 the changes are saved and ends up working

If you get an 400 internal error then that likely means the nginx config failed the test and will fail to start whenever you next restart the nginx-proxy-manager (at least that's what happened for me anyway). Could be something else erroring though ofc

[EricFROL]

Nono its actually working, weird tho.

[1A3Dev]

Nono its actually working, weird tho.

Well yeah like I said it worked fine for me until I restarted the proxy manager in which nginx failed to start and I had to manually go into the nginx configs and remove the lines that were causing it to break. That was apparently fixed though so might be a different issue you are having

[sylphrena0]

It looks like the last run failed and the helpful bot didn't post the reason, so some of those fixes might not be in the latest pr image. I will probably take a look this weekend, but y'all are welcome to poke around

The error is likely an attempt to call a nonexistent method to update the DNS records on creation/update of ddns records, so it would make sense that it would work on restart. This is fixed in the last commit, the build that is failing.

maybe I'll see if I can properly setup their dev environment so I don't need to wait for failed ci runs that I can't inspect to test lol

[jc21]

It looks like the last run failed and the helpful bot didn't post the reason, so some of those fixes might not be in the latest pr image. I will probably take a look this weekend, but y'all are welcome to poke around

The error is likely an attempt to call a nonexistent method to update the DNS records on creation/update of ddns records, so it would make sense that it would work on restart. This is fixed in the last commit, the build that is failing.

maybe I'll see if I can properly setup their dev environment so I don't need to wait for failed ci runs that I can't inspect to test lol

Here's what CI is saying for Cypress tests:

FullCertProvision.cy.js

10:58:52  cypress-1  | [Backend API] POST /api/tokens
10:58:52  cypress-1  | [Backend API] Axios Error: AxiosError: Request failed with status code 502
10:58:52  cypress-1  |   1) Full Certificate Provisions
10:58:52  cypress-1  |        "before all" hook for "Should be able to create new http certificate":
10:58:52  cypress-1  |      CypressError: `cy.task('backendApiPost')` failed with the following error:
10:58:52  cypress-1  | 
10:58:52  cypress-1  |     data: '<html>\r\n' +
10:58:52  cypress-1  |       '<head><title>502 Bad Gateway</title></head>\r\n' +
10:58:52  cypress-1  |       '<body>\r\n' +
10:58:52  cypress-1  |       '<center><h1>502 Bad Gateway</h1></center>\r\n' +
10:58:52  cypress-1  |       '<hr><center>openresty</center>\r\n' +
10:58:52  cypress-1  |       '</body>\r\n' +
10:58:52  cypress-1  |       '</html>\r\n'
10:58:52  cypress-1  |   },
10:58:52  cypress-1  | > Request failed with status code 502

there's a few of them applying to the streams tests too.

Here's the app log error related to it. When this uncaught error happens, the process dies and restarts again, causing the 502.

[90m[12:57:47 AM][39m [90m[DDNS     ][39m [90m›[39m [34mℹ [39m [34m[4minfo[24m     [39m Checking for DDNS updates...
Uncaught TypeError: internalAccessList.getAll is not a function

FROM
Object._getAccessLists (/app/lib/ddns_resolver/ddns_updater.js:152:29)
Timeout.updateDynamicDnsRecords [as _onTimeout] (/app/lib/ddns_resolver/ddns_updater.js:56:23)
listOnTimeout (node:internal/timers:581:17)

[jc21]

You can spin up a dev setup by running ./scripts/start-dev and ./scripts/destroy-dev to bring it all down.