Let's Encrypt Issue

[Andrewm30]

Describe your issue

Unable to get Let's Encrypt to work. I have the URL using a public IP with nothing else on it. Verified ports 80 and 443 are open via openport checker, and I have the correct firewall enabled on my fortigate for Central and DNAT (incoing on the public, outgoing on the public is correct). At one point I had a staging cert but it never would give me a real one. This was last Thurs/Friday.

Letsdebug says that everything is good to go on the URL I'm exposing. A record in network solutions solves properly.

Server Software (please complete the following information):

• OS: [e.g. Ubuntu] Windows (tried ubuntu as well)
• Virtualization: [e.g. Docker] Xen
• Network: [e.g. LAN/WAN, reverse proxy, cloudflare, ssl offload, etc...] WAN/LAN accessible
• Version: [e.g. 1.0.43] 1.0.74
• Node: [e.g. 18.4.0] Latest

Your config.json file

{
  "$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
  "__comment1__": "This is a simple configuration file, all values and sections that start with underscore (_) are ignored. Edit a section and remove the _ in front of the name. Refer to the user's guide for details.",
  "__comment2__": "See node_modules/meshcentral/sample-config-advanced.json for a more advanced example.",
  "settings": {
    "cert": "siotekremote.siotek.net",
    "_WANonly": true,
    "_LANonly": true,
    "_sessionKey": "MyReallySecretPassword1",
    "port": 443,
    "_aliasPort": 443,
    "redirPort": 80,
    "_redirAliasPort": 80
  },
  "domains": {
    "": {
      "_title": "MyServer",
      "_title2": "Servername",
      "_minify": true,
      "_newAccounts": true,
      "_userNameIsEmail": true
    }
  },
  "letsencrypt": {
    "__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before trying Let's Encrypt.",
    "email": "[email protected]",
    "names": "siotekremote.siotek.net",
    "skipChallengeVerification": false,    _I added this note when I pasted this - I saw this suggested somewhere on the forum
    "production": false
  }
}

Le/Leevents

> le
{
    "configOk": true,
    "leDomains": [
        "siotekremote.siotek.net"
    ],
    "challenges": {
        "RGG2Ucs-qxS5OCBfEETCjdJLRXm7wdWxhDhat4fq6as": "RGG2Ucs-qxS5OCBfEETCjdJLRXm7wdWxhDhat4fq6as.IAm9DxW5_hLBBNcZ7sTOTVI5AICxBIeWZHo-xo5yTi0"
    },
    "production": false,
    "webServer": true,
    "certPath": "c:\\Mesh Central\\meshcentral-data\\letsencrypt-certs",
    "skipChallengeVerification": false,
    "cert": "None"
}
> leevents
8/29/2022 10:03:01 AM - Getting certs from local store (Staging)
8/29/2022 10:03:01 AM - No certificate files found
8/29/2022 10:03:08 AM - Got no certificates, asking for one now.
8/29/2022 10:03:08 AM - Generating private key...
8/29/2022 10:03:08 AM - Setting up ACME client...
8/29/2022 10:03:08 AM - Creating certificate request...
8/29/2022 10:03:08 AM - Requesting certificate from Let's Encrypt...

[Andrewm30]

Minor correct to the above, I was on version 1.0.72 - updated to 1.0.74

Now leevents is giving the following:

leevents 8/29/2022 10:06:18 AM - Getting certs from local store (Staging) 8/29/2022 10:06:18 AM - No certificate files found 8/29/2022 10:06:23 AM - Got no certificates, asking for one now. 8/29/2022 10:06:23 AM - Generating private key... 8/29/2022 10:06:24 AM - Setting up ACME client... 8/29/2022 10:06:24 AM - Creating certificate request... 8/29/2022 10:06:24 AM - Requesting certificate from Let's Encrypt... 8/29/2022 10:13:31 AM - Failed to obtain certificate: connect ETIMEDOUT 128.136.251.93:80 8/29/2022 10:17:00 AM - Failed to respond to challenge, token: letsdebug-test, table: {}. 8/29/2022 10:17:00 AM - Failed to respond to challenge, token: URg7AP6ZUtl5BU3EwQl5SSbSoNWWUMVqccPzpr0SYQ0, table: {}. 8/29/2022 10:17:00 AM - Failed to respond to challenge, token: URg7AP6ZUtl5BU3EwQl5SSbSoNWWUMVqccPzpr0SYQ0, table: {}. 8/29/2022 10:17:00 AM - Failed to respond to challenge, token: URg7AP6ZUtl5BU3EwQl5SSbSoNWWUMVqccPzpr0SYQ0, table: {}. 8/29/2022 10:17:00 AM - Failed to respond to challenge, token: URg7AP6ZUtl5BU3EwQl5SSbSoNWWUMVqccPzpr0SYQ0, table: {}.

However, port 80 is open to the box, IIS is not installed, - and letsdebug from the windows box directly says its good to go.

Perhaps I'm missing something is using port 80? THis is a fresh windows 2019 install with only mesh central on it.

[Andrewm30]

netstat -aon | findstr :80 and 443

The only thing listening on port 80/443 is node.js

Sorry for all the updates, just wanted to give what information I have.

[Ylianst]

I am stumped. It indeed looks like you are ready to go, especially since letsdebug did some tests on your server and it worked.

As you correctly understand, Let's Encrypt needs to be able to resolve your DNS name from different locations around the world and be able to connect to your server on port 80. It looks like that is working for you.

The only thing I could think of is that prior to having this all working, you tried to get a Let's Encrypt certificate many times without the right setup and got placed on the Let's Encrypt "hold" list. This is when you try to many times and Let's Encrypt will not talk to your IP address for a time.

For port 80, I use the mobile phone or something completely outside my network and access my server using HTTP/80 and see if I get redirected to HTTPS/443. If that works, it's a good sign port 80 is available externally.

[si458]

what gateway do you have?

what happens if you visit the external ip address in your browser on the server itself? http://128.136.251.93

as this looks like a port forwarding issue, it shows it cant connect to itself from itself 8/29/2022 10:13:31 AM - Failed to obtain certificate: connect ETIMEDOUT 128.136.251.93:80

i know on our mikrotik router we have to put a masquarade rule in so any device internally trying to access its own external ip address gets forwarded correctly

you could also try setting skipChallengeVerification: true in your config.json to skip the verifying bit

[PathfinderNetworks]

Do you have any threat prevention modules on your router/firewall? I use Untangle and had recently activated the Threat Prevention module. The Threat Prevention database that Untangle uses is flagging most of the LetsEncrypt IP addresses as 'High Threat' and is blocking them from connecting. I had to create a bypass rule to allow LetsEncrypt access to the /.well-known/acme-challenge/ URI before my certificates could be renewed.

[PathfinderNetworks]

BTW, if I try to access the URL you have listed in your config above, siotekremote.siotek.net, I get a 'connection refused' message. Which means it is actively blocking me from connecting.

[Andrewm30]

Right now I have them disabled as I'm troubleshooting this.

I have a fortigate 101E - I have SNAT enabled. (central nating).

In/out for .93 on my public

If ytou check openports, 80 and 443 is open.

It is NAT'd correctly.

The odd part is, until I enabled ignore challenge verification to true as a test as above, everything was working and resolving as intended, once I enabled it, it broke it. Now with it DISabled, it still won't start the server now so I'm messing with the config file.

[Andrewm30]

So removing that line completely, and rebooting fixed the issue - restarting the service did not fix it for some reason.

its accessible from the outside, tested on cell phone.

[Ylianst]

One idea would be to run CertBot and see if you can get a staining Let's Encrypt cert. This would eliminate MeshCentral as the issue, but if it does, there is still a open question as to why this is not working.

[tradexsrl]

hi , i have a similar problem after updating from 1.0.74 to 1.0.75 , now firefox it says the site is not secured and cannot access it. i deleted the cache and the cookie now i get the same warning but i can access adding an exception. looking at the security in the url it say the connection is not secured and there is no info on certificate. it seems there is a problem on letsencrypt client , please let me know.

[tradexsrl]

weird...after a meshcentral service restart , letsencrypt start working again

[si458]

weird...after a meshcentral service restart , letsencrypt start working again

i know the is a delay with the lets encrypt certificates being applied, i found this out, it was issuing a self signed cert but then after 60 seconds, the LE certificate was applied (i proved this by checking the node logs, it shows self signed then after about 30 secs i renewed the LE cert and then restarted itself to apply it)

[Andrewm30]

So, I've got some good and bad news

I installed certbot, ran cerbot certonly --standalone --staging (which Im actually unsure that staging part worked), and it failed to grab a cert at first due to not being able to bind to port 80

I then stopped meshcentral, so it was bound and using port 80 (nothing else was using it)

Re-ran the above, and it worked - it generated the cert files in the certbot folder, siotekremote.siotek.net

Please enter the domain name(s) you would like on your certificate (comma and/or space separated) (Enter 'c' to cancel): siotekremote.siotek.net Requesting a certificate for siotekremote.siotek.net

Successfully received certificate. Certificate is saved at: C:\Certbot\live\siotekremote.siotek.net\fullchain.pem Key is saved at: C:\Certbot\live\siotekremote.siotek.net\privkey.pem This certificate expires on 2022-11-28.

So, it does appear to be an issue with meshcentral. The problem is, I'm having this issue in both a windows environment, and a linux Ubuntu. Neither will grab a cert. It doesn't seem to know how to answer the challenge question properly.

Not sure what to do from here

am I able to use the cerbot certification and drop it into the folder in meshcentral let's encrypt? Its the cert, chain, private and fullchain files.

[si458]

@Andrewm30 this still seems to me like an issue with your firewall because certbot doesnt check the url first, it asks letsencrypt to check instead, where as meshcentral checks the url first before asking letsencrypt to check

on your ubuntu pc running the meshcentral server, can you do curl -v --insecure https://(DNSNAMEOFTHEUBUNTUSERVER) and show us the reply? if this times out, its a firewall issue, if not its something else

[JReally]

I experienced the same as @tradexsrl except that I rolled back to v59 and everything worked again.

JR

[Andrewm30]

I solved this issue.

I never could make this work on the windows box. Added an entry into the host file for the internal IP to be resolved. These PCs weren't on a domain - I think adding that piece into the install guide would help eliminate some of these questions (just a suggestion, that the box has to resolve the internal IP for the lets encrypt to work). However, Certbot does not need this to function, it works without that, as mentioned above.

I have it running in debian now without issue. The windows version would just randomly stop working for no reason. I would make a minor change to the config file, for example, disabling nad enabling lets encrypt, and it would simply never start again. Even restoring a clean, unaltered version of the ocnfig file - it would never listen on port 80 or 443 again - had this happen multiple times and was forced to reboot - just an FYI.

So this was actually a DNS issue with lets encrypt and not a firewall prot forwarding issue

Thanks all for the replies - I appreciate it.

[dinger1986]

if this is solved can you close it please :)