bunkerweb icon indicating copy to clipboard operation
bunkerweb copied to clipboard

[FEATURE] Behavior on reboot (hard reboot)

Open svmrw opened this issue 3 months ago • 2 comments

What happened?

I decided to check how my entire configured system would work in case of server failure and subsequent recovery. I have done a hard reboot of my VPS several times.

Here are the problems I found:

  1. The first hard reboot seems to go well, but I found that an invalid certificate from example.org is issued first, while a valid one from letsencrypt is in the cache, which is then loaded. Perhaps it is worth somehow loading the previously issued and valid one right away. But this is not a serious problem, unlike the second one.

  2. If you do a hard reboot right after the first one, bunkerweb won't load at all. Below are the logs. In addition to the logs, you can see that /bin/bash ./entrypoint.sh loads one CPU core by 100%. There are two possible outcomes in the logs. The first one cannot load the certificate, and the second one is unclear what the matter is. I copied everything that was there. I didn't add or subtract anything in the last lines.

How to reproduce?

Perform a forced hard reset several times.

docker compose up -d --force-recreate solves the problem, but there may be a situation when it is impossible to do this and the service will not work until this command is executed.

A hard and unsafe reboot must be performed.

Configuration file(s) (yaml or .env)

Same as here, plus syslog is the same as in the documentation:
https://github.com/bunkerity/bunkerweb/blob/master/misc/integrations/docker.mariadb.yml
https://github.com/bunkerity/bunkerweb/issues/2461
https://docs.bunkerweb.io/latest/web-ui/#syslog-ng-configuration

Relevant log output

bunkerweb service:
[2025-09-01 23:48:03 +0500] [GENERATOR] [12] [ℹ️ ] - Removing old files ...
[2025-09-01 23:48:03 +0500] [GENERATOR] [12] [ℹ️ ] - Rendering templates ...
[2025-09-01 23:48:04 +0500] [GENERATOR] [12] [ℹ️ ] - Generator successfully executed !
[2025-09-01 23:48:04] - ENTRYPOINT - ℹ️ - Starting nginx ...
2025/09/01 23:48:04 [emerg] 29#29: cannot load certificate "/var/cache/bunkerweb/misc/default-server-cert.pem": PEM_read_bio_X509_AUX() failed (SSL: error:0480006C:PEM routines::no start line:Expecting: TRUSTED CERTIFICATE)
nginx: [emerg] cannot load certificate "/var/cache/bunkerweb/misc/default-server-cert.pem": PEM_read_bio_X509_AUX() failed (SSL: error:0480006C:PEM routines::no start line:Expecting: TRUSTED CERTIFICATE)


bw-scheduler service:
[2025-09-01 23:48:54 +0500] [SCHEDULER.HEALTHCHECK] [38] [⚠️ ] - Can't send API request to http://bunkerweb:5000/health : Request failed: HTTPConnectionPool(host='bunkerweb', port=5000): Max retries exceeded with url: /health (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7c512c44fc50>: Failed to establish a new connection: [Errno 111] Connection refused')), healthcheck will be retried in 30 seconds ...
[2025-09-01 23:49:24 +0500] [SCHEDULER.HEALTHCHECK] [38] [⚠️ ] - Can't send API request to http://bunkerweb:5000/health : Request failed: HTTPConnectionPool(host='bunkerweb', port=5000): Max retries exceeded with url: /health (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7c512b7d82b0>: Failed to establish a new connection: [Errno 111] Connection refused')), healthcheck will be retried in 30 seconds ...
[2025-09-01 23:49:54 +0500] [SCHEDULER.HEALTHCHECK] [38] [⚠️ ] - Can't send API request to http://bunkerweb:5000/health : Request failed: HTTPConnectionPool(host='bunkerweb', port=5000): Max retries exceeded with url: /health (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7c512b7d82b0>: Failed to establish a new connection: [Errno 111] Connection refused')), healthcheck will be retried in 30 seconds ...


==========

bunkerweb service:
[2025-09-02 00:18:49 +0500] [GENERATOR] [11] [ℹ️ ] - Settings : /usr/share/bunkerweb/settings.json
[2025-09-02 00:18:49 +0500] [GENERATOR] [11] [ℹ️ ] - Templates : /usr/share/bunkerweb/confs
[2025-09-02 00:18:49 +0500] [GENERATOR] [11] [ℹ️ ] - Core : /usr/share/bunk
6
stdo


bw-scheduler service:
[2025-09-02 00:18:58 +0500] [SCHEDULER] [38] [ℹ️ ] - Removing old custom configs files ...
[2025-09-02 00:18:58 +0500] [SCHEDULER] [38] [ℹ️ ] - Sending /etc/bunkerweb/configs to all reachable BunkerWeb instances ...
[2025-09-02 00:18:58 +0500] [Ag
stdoutʧ?ַ?ϰS[2025-09-02 00:19:33] - ENTRYPOINT - ℹ️ - Starting the job sch

Conclusion

I would like everything to recover after a server failure. Check for yourself how it handles such hard reboots to simulate server crashes. I'll repeat again - perform several hard reboots. It recovered and a second failure is needed right away. I always got errors on the second one.

BunkerWeb version

1.6.4

What integration are you using?

Docker

Linux distribution (if applicable)

Ubuntu 24.04

Removed private data

  • [x] I have removed all private data from the configuration file and the logs

Code of Conduct

  • [x] I agree to follow this project's Code of Conduct

svmrw avatar Sep 01 '25 19:09 svmrw

Hi @svmrw, this is a normal behavior but I guess we could add a setting that tells BunkerWeb to keep it's original configuration when starting

TheophileDiot avatar Sep 02 '25 07:09 TheophileDiot

@TheophileDiot, Well, the logs I provided hardly look like normal behavior. Please check everything yourself. This happens precisely during the second hard reboot. I understand that this is unlikely to happen on the server, but there is such a possibility.

svmrw avatar Sep 02 '25 07:09 svmrw