Certbot breaks on RHEL with new Apache packages

[ViktorNova]

I don't need help, but wanted to report a bug:

My operating system is (include version):

CentOS 8, with Apache 2.4.48 from Remi repo

I installed Certbot with (snap, OS package manager, pip, certbot-auto, etc):

Snap, certbot 1.17.0

I ran this command and it produced this output:

certbot --apache

Error in checking parameter list: 
Certbot doesn't know how to automatically configure the web server on this system. However, it can still get a certificate for you. Please run "certbot certonly" to do so. You'll need to manually configure your web server to use the resulting certificate.

Certbot's behavior differed from what I expected because:

There is no Apache error, and mod_ssl is installed and working just fine

Here is a Certbot log showing the issue (if available):

Logs are stored in /var/log/letsencrypt by default. Feel free to redact domains, e-mail and IP addresses as you see fit.

2021-07-15 17:51:03,735:DEBUG:urllib3.connectionpool:http://localhost:None "GET /v2/connections?snap=certbot&interface=content HTTP/1.1" 200 97
2021-07-15 17:51:04,433:DEBUG:certbot._internal.main:certbot version: 1.17.0
2021-07-15 17:51:04,433:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/1280/bin/certbot
2021-07-15 17:51:04,433:DEBUG:certbot._internal.main:Arguments: ['--apache', '--non-interactive', '--agree-tos', '-m', '[email protected]', '-d', 'website.com,www.website.com', '--preconfigured-renewal']
2021-07-15 17:51:04,433:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2021-07-15 17:51:04,446:DEBUG:certbot._internal.log:Root logging level set at 30
2021-07-15 17:51:04,447:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2021-07-15 17:51:04,595:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.48
2021-07-15 17:51:04,627:WARNING:certbot_apache._internal.apache_util:Error in checking parameter list: 
2021-07-15 17:51:04,627:DEBUG:certbot._internal.plugins.disco:Misconfigured PluginEntryPoint#apache: Apache is unable to check whether or not the module is loaded because Apache is misconfigured.
Traceback (most recent call last):
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot/_internal/plugins/disco.py", line 158, in prepare
    self._initialized.prepare()  # type: ignore
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot_apache/_internal/configurator.py", line 366, in prepare
    self.parser = self.get_parser()
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot_apache/_internal/override_centos.py", line 81, in get_parser
    return CentOSParser(
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot_apache/_internal/override_centos.py", line 169, in __init__
    super().__init__(*args, **kwargs)
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot_apache/_internal/parser.py", line 67, in __init__
    self.update_runtime_variables()
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot_apache/_internal/override_centos.py", line 174, in update_runtime_variables
    super().update_runtime_variables()
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot_apache/_internal/parser.py", line 278, in update_runtime_variables
    self.update_defines()
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot_apache/_internal/parser.py", line 285, in update_defines
    self.variables = apache_util.parse_defines(self.configurator.options.ctl)
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot_apache/_internal/apache_util.py", line 149, in parse_defines
    matches = parse_from_subprocess(define_cmd, r"Define: ([^ \n]*)")
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot_apache/_internal/apache_util.py", line 209, in parse_from_subprocess
    stdout = _get_runtime_cfg(command)
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot_apache/_internal/apache_util.py", line 242, in _get_runtime_cfg
    raise errors.MisconfigurationError(
certbot.errors.MisconfigurationError: Apache is unable to check whether or not the module is loaded because Apache is misconfigured.
2021-07-15 17:51:04,632:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_centos.CentOSConfigurator object at 0x7f09ce2d2550>
Prep: Apache is unable to check whether or not the module is loaded because Apache is misconfigured.
2021-07-15 17:51:04,633:DEBUG:certbot._internal.plugins.selection:Selected authenticator None and installer None

Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:

The Vhost is very basic and works great with Certbot on Apache 2.4.37

---

I believe this is a bug in Certbot. Apache 2.4.4+ is deprecating the apachectl command, it can no longer be used for anything other than stopping/starting/restarting httpd

Solution for now is downgrade Apache to 2.37 (default CentOS 8 version) which still works with Certbot: dnf downgrade httpd

[bmw]

Due to the large variations in how httpd is configured by different distros, the way our Apache plugin works is it determines the OS you are running on and then defaults to using the Apache conventions for that distro. In your case, that doesn't work because you're installing httpd from another source than the CentOS 8 repositories.

I think you should be able to make things work by setting some command line values such as --apache-ctl httpd. Does that work for you?

[ViktorNova]

@bmw I actally tried that very thing, since many of apachectl's commands are available to httpd. But the syntax is slightly different and this produced an error - in this case it choked on httpd configtest since that isn't a valid command on httpd Here's the log from that

Log output:

2021-07-16 03:58:40,820:DEBUG:urllib3.connectionpool:http://localhost:None "GET /v2/connections?snap=certbot&interface=content HTTP/1.1" 200 97
2021-07-16 03:58:41,055:DEBUG:certbot._internal.main:certbot version: 1.17.0
2021-07-16 03:58:41,056:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/1280/bin/certbot
2021-07-16 03:58:41,056:DEBUG:certbot._internal.main:Arguments: ['--apache', '--apache-ctl=/usr/sbin/httpd', '--preconfigured-renewal']
2021-07-16 03:58:41,056:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2021-07-16 03:58:41,068:DEBUG:certbot._internal.log:Root logging level set at 30
2021-07-16 03:58:41,069:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2021-07-16 03:58:41,101:ERROR:certbot.util:Error while running /usr/sbin/httpd configtest.

Usage: /usr/sbin/httpd [-D name] [-d directory] [-f file]
                       [-C "directive"] [-c "directive"]
                       [-k start|restart|graceful|graceful-stop|stop]
                       [-v] [-V] [-h] [-l] [-L] [-t] [-T] [-S] [-X]
Options:
  -D name            : define a name for use in <IfDefine name> directives
  -d directory       : specify an alternate initial ServerRoot
  -f file            : specify an alternate ServerConfigFile
  -C "directive"     : process directive before reading config files
  -c "directive"     : process directive after reading config files
  -e level           : show startup errors of level (see LogLevel)
  -E file            : log startup errors to file
  -v                 : show version number
  -V                 : show compile settings
  -h                 : list available command line options (this page)
  -l                 : list compiled in modules
  -L                 : list available configuration directives
  -t -D DUMP_VHOSTS  : show parsed vhost settings
  -t -D DUMP_RUN_CFG : show parsed run settings
  -S                 : a synonym for -t -D DUMP_VHOSTS -D DUMP_RUN_CFG
  -t -D DUMP_MODULES : show all loaded modules 
  -M                 : a synonym for -t -D DUMP_MODULES
  -t -D DUMP_INCLUDES: show all included configuration files
  -t                 : run syntax check for config files
  -T                 : start without DocumentRoot(s) check
  -X                 : debug mode (only one worker, do not detach)

2021-07-16 03:58:41,102:DEBUG:certbot._internal.plugins.disco:Misconfigured PluginEntryPoint#apache: Error while running /usr/sbin/httpd configtest.

Usage: /usr/sbin/httpd [-D name] [-d directory] [-f file]
                       [-C "directive"] [-c "directive"]
                       [-k start|restart|graceful|graceful-stop|stop]
                       [-v] [-V] [-h] [-l] [-L] [-t] [-T] [-S] [-X]
Options:
  -D name            : define a name for use in <IfDefine name> directives
  -d directory       : specify an alternate initial ServerRoot
  -f file            : specify an alternate ServerConfigFile
  -C "directive"     : process directive before reading config files
  -c "directive"     : process directive after reading config files
  -e level           : show startup errors of level (see LogLevel)
  -E file            : log startup errors to file
  -v                 : show version number
  -V                 : show compile settings
  -h                 : list available command line options (this page)
  -l                 : list compiled in modules
  -L                 : list available configuration directives
  -t -D DUMP_VHOSTS  : show parsed vhost settings
  -t -D DUMP_RUN_CFG : show parsed run settings
  -S                 : a synonym for -t -D DUMP_VHOSTS -D DUMP_RUN_CFG
  -t -D DUMP_MODULES : show all loaded modules 
  -M                 : a synonym for -t -D DUMP_MODULES
  -t -D DUMP_INCLUDES: show all included configuration files
  -t                 : run syntax check for config files
  -T                 : start without DocumentRoot(s) check
  -X                 : debug mode (only one worker, do not detach)
Traceback (most recent call last):
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot_apache/_internal/configurator.py", line 2468, in config_test
    util.run_script(self.options.conftest_cmd)
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot/util.py", line 115, in run_script
    raise errors.SubprocessError(msg)
certbot.errors.SubprocessError: Error while running /usr/sbin/httpd configtest.

Usage: /usr/sbin/httpd [-D name] [-d directory] [-f file]
                       [-C "directive"] [-c "directive"]
                       [-k start|restart|graceful|graceful-stop|stop]
                       [-v] [-V] [-h] [-l] [-L] [-t] [-T] [-S] [-X]
Options:
  -D name            : define a name for use in <IfDefine name> directives
  -d directory       : specify an alternate initial ServerRoot
  -f file            : specify an alternate ServerConfigFile
  -C "directive"     : process directive before reading config files
  -c "directive"     : process directive after reading config files
  -e level           : show startup errors of level (see LogLevel)
  -E file            : log startup errors to file
  -v                 : show version number
  -V                 : show compile settings
  -h                 : list available command line options (this page)
  -l                 : list compiled in modules
  -L                 : list available configuration directives
  -t -D DUMP_VHOSTS  : show parsed vhost settings
  -t -D DUMP_RUN_CFG : show parsed run settings
  -S                 : a synonym for -t -D DUMP_VHOSTS -D DUMP_RUN_CFG
  -t -D DUMP_MODULES : show all loaded modules 
  -M                 : a synonym for -t -D DUMP_MODULES
  -t -D DUMP_INCLUDES: show all included configuration files
  -t                 : run syntax check for config files
  -T                 : start without DocumentRoot(s) check
  -X                 : debug mode (only one worker, do not detach)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot/_internal/plugins/disco.py", line 158, in prepare
    self._initialized.prepare()  # type: ignore
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot_apache/_internal/configurator.py", line 348, in prepare
    self.config_test()
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot_apache/_internal/override_centos.py", line 49, in config_test
    super().config_test()
  File "/var/lib/snapd/snap/certbot/1280/lib/python3.8/site-packages/certbot_apache/_internal/configurator.py", line 2470, in config_test
    raise errors.MisconfigurationError(str(err))
certbot.errors.MisconfigurationError: Error while running /usr/sbin/httpd configtest.

Usage: /usr/sbin/httpd [-D name] [-d directory] [-f file]
                       [-C "directive"] [-c "directive"]
                       [-k start|restart|graceful|graceful-stop|stop]
                       [-v] [-V] [-h] [-l] [-L] [-t] [-T] [-S] [-X]
Options:
  -D name            : define a name for use in <IfDefine name> directives
  -d directory       : specify an alternate initial ServerRoot
  -f file            : specify an alternate ServerConfigFile
  -C "directive"     : process directive before reading config files
  -c "directive"     : process directive after reading config files
  -e level           : show startup errors of level (see LogLevel)
  -E file            : log startup errors to file
  -v                 : show version number
  -V                 : show compile settings
  -h                 : list available command line options (this page)
  -l                 : list compiled in modules
  -L                 : list available configuration directives
  -t -D DUMP_VHOSTS  : show parsed vhost settings
  -t -D DUMP_RUN_CFG : show parsed run settings
  -S                 : a synonym for -t -D DUMP_VHOSTS -D DUMP_RUN_CFG
  -t -D DUMP_MODULES : show all loaded modules 
  -M                 : a synonym for -t -D DUMP_MODULES
  -t -D DUMP_INCLUDES: show all included configuration files
  -t                 : run syntax check for config files
  -T                 : start without DocumentRoot(s) check
  -X                 : debug mode (only one worker, do not detach)

2021-07-16 03:58:41,105:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_centos.CentOSConfigurator object at 0x7fc364185160>
Prep: Error while running /usr/sbin/httpd configtest.

Usage: /usr/sbin/httpd [-D name] [-d directory] [-f file]
                       [-C "directive"] [-c "directive"]
                       [-k start|restart|graceful|graceful-stop|stop]
                       [-v] [-V] [-h] [-l] [-L] [-t] [-T] [-S] [-X]
Options:
  -D name            : define a name for use in <IfDefine name> directives
  -d directory       : specify an alternate initial ServerRoot
  -f file            : specify an alternate ServerConfigFile
  -C "directive"     : process directive before reading config files
  -c "directive"     : process directive after reading config files
  -e level           : show startup errors of level (see LogLevel)
  -E file            : log startup errors to file
  -v                 : show version number
  -V                 : show compile settings
  -h                 : list available command line options (this page)
  -l                 : list compiled in modules
  -L                 : list available configuration directives
  -t -D DUMP_VHOSTS  : show parsed vhost settings
  -t -D DUMP_RUN_CFG : show parsed run settings
  -S                 : a synonym for -t -D DUMP_VHOSTS -D DUMP_RUN_CFG
  -t -D DUMP_MODULES : show all loaded modules 
  -M                 : a synonym for -t -D DUMP_MODULES
  -t -D DUMP_INCLUDES: show all included configuration files
  -t                 : run syntax check for config files
  -T                 : start without DocumentRoot(s) check
  -X                 : debug mode (only one worker, do not detach)

2021-07-16 03:58:41,105:DEBUG:certbot._internal.plugins.selection:Selected authenticator None and installer None

I dug a little more into this though and according to Apache's documentation on apachectl, this behavior might indicate that the package I'm using from the Remi repo might be set up incorrectly - would you agree?

[ViktorNova]

(oops, I closed this by mistake)

[bmw]

Ah sorry for the bad suggestion. Thanks for providing that log though and doing a little digging yourself.

It looks to me like this is a patch Fedora/RHEL are applying to apachectl. It hasn't made it to RHEL directly yet, but it's there in Fedora and presumably has made it to the Remi repo. This code does not exist in apachectl upstream.

Our Apache plugin currently works on Fedora due to the way we set up parameters in the Fedora specific override. A user is currently unable to set things up this way themselves on the command line due to the code here which our Fedora specific override changes here.

Assuming this seems right to you, I'll leave this issue open as it'll need to be tackled before this change makes it's way to RHEL, presumably in RHEL 9, but to be honest, I doubt we'll be able to prioritize this before then.

[jguizarj]

Hi! a RHEL 9 user here

Certbot is not working with RHEL 9 :(

The upstream version of Apache for RHEL 9 is 2.4.51, I'm having the same issue as @ViktorNova

[gamesharkmike]

On RHEL9

Error in line 241 of /usr/local/lib/python3.9/site-packages/certbot_apache/_internal/apache_util.py

Trying to debug and get stderr, edited the section to:

    # Small errors that do not impede
    if proc.returncode != 0:
        #logger.warning("Error in checking parameter list: %s", stderr)
        print(stderr)
        #raise errors.MisconfigurationError(
        #stderr)
        #    "Apache is unable to check whether or not the module is "
        #    "loaded because Apache is misconfigured.")

No error given, asked for email address and to accept the terms. It seems I was able to obtain a certificate. I was moving lots of things around I synced from another server so I have to try and replicate on a different RHEL9 install.

[gamesharkmike]

I have been able to replicate this on multiple RHEL9. The Python site packages directory sometimes got overriden with one on a mounted read-only snapd squashfs file system. I had to create my own folder, copy the files there and do a bind mount over the read-only folder.

[alessandroprudencio]

Same problem this here.

• CentOS Stream release 9
• Apache/2.4.53 (CentOS Stream)
• certbot 1.29.0

My solution at the moment, is to ignore error and generate new certificate with the command:

sudo certbot certonly --standalone -d mydomain.com -m "[email protected]" --keep-until-expiring --redirect --no-eff-email --agree-tos

[sinall]

Same problem this here.

• CentOS Stream release 9
• Apache/2.4.53 (CentOS Stream)
• certbot 1.29.0

My solution at the moment, is to ignore error and generate new certificate with the command:

sudo certbot certonly --standalone -d mydomain.com -m "[email protected]" --keep-until-expiring --redirect --no-eff-email --agree-tos

How to make use of those certificates with apache?

[StratsScorpio]

My solution is:

1. shutdown apache
2. copy config and edit config /etc/letsencrypt/renewal/domainname.default.com.conf
3. change authenticator = apache to authenticator = standalone
4. remove installer line; save
5. execute certbot renew
6. start apache
7. restore config

[github-actions[bot]]

We've made a lot of changes to Certbot since this issue was opened. If you still have this issue with an up-to-date version of Certbot, can you please add a comment letting us know? This helps us to better see what issues are still affecting our users. If there is no activity in the next 30 days, this issue will be automatically closed.

[github-actions[bot]]

This issue has been closed due to lack of activity, but if you think it should be reopened, please open a new issue with a link to this one and we'll take a look.