wazuh-docker
wazuh-docker copied to clipboard
wazuh
4.5.4 - Dashboard and Indexer containers keeps rebooting
Hello,
I discovered recently this project and wanted to test it but unfortunately the dashboard and the indexer containers are rebooting in loop.
Here are the logs for the dashboard :
2023-10-24T01:07:57.752329704Z An OpenSearch Dashboards keystore already exists. Overwrite? [y/N] Created OpenSearch Dashboards keystore in /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore
2023-10-24T01:07:59.164926576Z grep: /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml: Permission denied
2023-10-24T01:07:59.165989719Z /wazuh_app_config.sh: line 53: /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml: Permission denied
2023-10-24T01:07:59.485812200Z internal/fs/utils.js:332
2023-10-24T01:07:59.485990451Z throw err;
2023-10-24T01:07:59.486024895Z ^
2023-10-24T01:07:59.486050019Z
2023-10-24T01:07:59.486075522Z Error: EACCES: permission denied, open '/usr/share/wazuh-dashboard/config/opensearch_dashboards.yml'
2023-10-24T01:07:59.486113930Z at Object.openSync (fs.js:498:3)
2023-10-24T01:07:59.486144565Z at Object.readFileSync (fs.js:394:35)
2023-10-24T01:07:59.486176387Z at readYaml (/usr/share/wazuh-dashboard/node_modules/@osd/apm-config-loader/target/utils/read_config.js:37:52)
2023-10-24T01:07:59.486213555Z at Object.exports.getConfigFromFiles (/usr/share/wazuh-dashboard/node_modules/@osd/apm-config-loader/target/utils/read_config.js:62:22)
2023-10-24T01:07:59.486250789Z at exports.loadConfiguration (/usr/share/wazuh-dashboard/node_modules/@osd/apm-config-loader/target/config_loader.js:43:38)
2023-10-24T01:07:59.486284344Z at module.exports (/usr/share/wazuh-dashboard/src/apm.js:58:15)
2023-10-24T01:07:59.486316443Z at Object.<anonymous> (/usr/share/wazuh-dashboard/src/cli/dist.js:32:18)
2023-10-24T01:07:59.486355195Z at Module._compile (internal/modules/cjs/loader.js:1085:14)
2023-10-24T01:07:59.486388762Z at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
2023-10-24T01:07:59.486423607Z at Module.load (internal/modules/cjs/loader.js:950:32) {
2023-10-24T01:07:59.486453665Z errno: -13,
2023-10-24T01:07:59.486479502Z syscall: 'open',
2023-10-24T01:07:59.486505558Z code: 'EACCES',
2023-10-24T01:07:59.486536593Z path: '/usr/share/wazuh-dashboard/config/opensearch_dashboards.yml'
2023-10-24T01:07:59.486568123Z }
And the logs for the indexer :
2023-10-24T01:07:46.654014811Z grep: /usr/share/wazuh-indexer/opensearch.yml: Permission denied
2023-10-24T01:07:46.661009616Z grep: /usr/share/wazuh-indexer/opensearch.yml: Permission denied
2023-10-24T01:07:49.841746364Z Exception in thread "main" SettingsException[Failed to load settings from /usr/share/wazuh-indexer/opensearch.yml]; nested: AccessDeniedException[/usr/share/wazuh-indexer/opensearch.yml];
2023-10-24T01:07:49.850778534Z at org.opensearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:98)
2023-10-24T01:07:49.850956040Z at org.opensearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:118)
2023-10-24T01:07:49.851002101Z at org.opensearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:109)
2023-10-24T01:07:49.851039433Z at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
2023-10-24T01:07:49.851075737Z at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
2023-10-24T01:07:49.851208806Z at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
2023-10-24T01:07:49.851249128Z at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
2023-10-24T01:07:49.851281970Z at org.opensearch.cli.Command.main(Command.java:101)
2023-10-24T01:07:49.851314398Z at org.opensearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:56)
2023-10-24T01:07:49.851349448Z Caused by: java.nio.file.AccessDeniedException: /usr/share/wazuh-indexer/opensearch.yml
2023-10-24T01:07:49.851389723Z at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
2023-10-24T01:07:49.851427301Z at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
2023-10-24T01:07:49.851460544Z at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
2023-10-24T01:07:49.851497734Z at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
2023-10-24T01:07:49.851531927Z at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
2023-10-24T01:07:49.851563877Z at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
2023-10-24T01:07:49.851600805Z at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
2023-10-24T01:07:49.851637116Z at java.base/java.nio.file.Files.newInputStream(Files.java:160)
2023-10-24T01:07:49.851669389Z at org.opensearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1111)
2023-10-24T01:07:49.851708511Z at org.opensearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:96)
2023-10-24T01:07:49.851742518Z ... 8 more
2023-10-24T01:07:51.109626175Z WARNING: A terminally deprecated method in java.lang.System has been called
2023-10-24T01:07:51.110208329Z WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.6.0.jar)
2023-10-24T01:07:51.110309844Z WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
2023-10-24T01:07:51.110358585Z WARNING: System::setSecurityManager will be removed in a future release
2023-10-24T01:07:52.741238869Z Exception in thread "main" SettingsException[Failed to load settings from /usr/share/wazuh-indexer/opensearch.yml]; nested: AccessDeniedException[/usr/share/wazuh-indexer/opensearch.yml];
2023-10-24T01:07:52.741837035Z at org.opensearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:98)
2023-10-24T01:07:52.741951331Z at org.opensearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:118)
2023-10-24T01:07:52.742006459Z at org.opensearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:109)
2023-10-24T01:07:52.742043200Z at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
2023-10-24T01:07:52.742091833Z at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
2023-10-24T01:07:52.742144681Z at org.opensearch.cli.Command.main(Command.java:101)
2023-10-24T01:07:52.742180794Z at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
2023-10-24T01:07:52.742224043Z at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
2023-10-24T01:07:52.742257996Z Caused by: java.nio.file.AccessDeniedException: /usr/share/wazuh-indexer/opensearch.yml
2023-10-24T01:07:52.742312880Z at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
2023-10-24T01:07:52.744432573Z at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
2023-10-24T01:07:52.744588344Z at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
2023-10-24T01:07:52.744628574Z at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
2023-10-24T01:07:52.744665415Z at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
2023-10-24T01:07:52.744702498Z at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
2023-10-24T01:07:52.744737530Z at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
2023-10-24T01:07:52.744777393Z at java.base/java.nio.file.Files.newInputStream(Files.java:160)
2023-10-24T01:07:52.744811279Z at org.opensearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1111)
2023-10-24T01:07:52.744864900Z at org.opensearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:96)
2023-10-24T01:07:52.744922436Z ... 7 more
All I did was to change the output ports 514:514 and 443:5601
Thank you in advance.
@Pestage Hi! What distribution do you use? I mean - what type of installation of wazuh you sticked to? Also if it's a Kubernetes cluster - what is the type and version of it?
Hi, I used Docker to install Wazuh under a Synology Host (NAS) (There is no Kubernetes cluster)
I finally test on a VPS with the Ubuntu install and it's working fine but it's just a trial VPS. (except that Windows vulnerabilities seems to be not displayed but it's another story)
I would like to use Docker so it will be free and hosted on my NAS. Ports 514 and 443 are already used on my docker host so maybe it is the issue.
I just tried to run it with 4.7.0 and got the same error, running from the single-node folder
@jay-oswald Hi! What are the precise steps to reproduce the issue? Just docker-compose up -d on local machine?
@jay-oswald Hi! What are the precise steps to reproduce the issue? Just docker-compose up -d on local machine?
Here's all the steps I followed, server is unraid using a Docker compose plugin.
Cloned the repo on my laptop. Checked out the v4.7.0 tag Scp the single node folder to the sever Ran the Docker compose script to generate certs Ran the main Docker compose script
The only modification I have made is changed the port for the dashboard, since 443 is used by my reverse proxy.
I checked the compose file and it's set the yaml and ranamed it in the volumes, and that file is at the correct path locally.
It could potentially be permissions issues? Part of my debugging I set every file (except the certs) to 644 and it didn't change anything
The auctal error I get is that it can't read the file, not a perms error
% git clone https://github.com/wazuh/wazuh-docker.git
% cd wazuh-docker
% git checkout v4.7.0
Note: switching to 'v4.7.0'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at dcf4842 Merge pull request #1129 from wazuh/chenge_revision_number
% cd single-node
% docker-compose -f generate-indexer-certs.yml run --rm generator
[+] Creating 1/1
✔ Network single-node_default Created 0.2s
[+] Running 5/5
✔ generator 4 layers [⣿⣿⣿⣿] 0B/0B Pulled 13.1s
✔ edaedc954fb5 Pull complete 8.7s
✔ 573f4d11a520 Pull complete 10.1s
✔ 8f200922197d Pull complete 10.1s
✔ 55a86de68c5c Pull complete 10.1s
The tool to create the certificates exists in the in Packages bucket
02/12/2023 09:25:06 INFO: Admin certificates created.
02/12/2023 09:25:06 INFO: Wazuh indexer certificates created.
02/12/2023 09:25:06 INFO: Wazuh server certificates created.
02/12/2023 09:25:06 INFO: Wazuh dashboard certificates created.
Moving created certificates to the destination directory
Changing certificate permissions
Setting UID indexer and dashboard
Setting UID for wazuh manager and worker
cp: cannot create regular file '/certificates/root-ca-manager.pem': Operation not permitted
cp: cannot create regular file '/certificates/root-ca-manager.key': Operation not permitted
chown: cannot access '/certificates/root-ca-manager.pem': No such file or directory
chown: cannot access '/certificates/root-ca-manager.key': No such file or directory
% docker compose up -d
[+] Running 43/3
✔ wazuh.dashboard 11 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 226.1s
✔ wazuh.manager 16 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 59.6s
✔ wazuh.indexer 13 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 243.8s
[+] Running 14/10
✔ Volume "single-node_wazuh-dashboard-config" Created 0.1s
✔ Volume "single-node_wazuh_agentless" Created 0.0s
✔ Volume "single-node_wazuh_api_configuration" Created 0.0s
✔ Volume "single-node_wazuh_var_multigroups" Created 0.0s
✔ Volume "single-node_filebeat_etc" Created 0.0s
✔ Volume "single-node_wazuh-indexer-data" Created 0.0s
✔ Volume "single-node_wazuh_queue" Created 0.0s
✔ Volume "single-node_wazuh_etc" Created 0.0s
✔ Volume "single-node_wazuh-dashboard-custom" Created 0.0s
✔ Volume "single-node_wazuh_active_response" Created 0.0s
✔ Volume "single-node_filebeat_var" Created 0.0s
✔ Volume "single-node_wazuh_logs" Created 0.0s
✔ Volume "single-node_wazuh_integrations" Created 0.0s
✔ Volume "single-node_wazuh_wodles" Created 0.0s
⠋ Container single-node-wazuh.indexer-1 Creating 0.0s
⠋ Container single-node-wazuh.manager-1 Creating 0.0s
Error response from daemon: No such image: wazuh/wazuh-manager:4.7.0
During this process I noticed three things:
- the errors when certificate generation, but the PEM files are present
- very long time to load images... Wazuh.indexeer is 1.5 GB large !
- There is no image for wazuh/wazuh-manager:4.7.0
p.s. please discard the last observation - probably it was an issue with docker desktop running on macbook, as it collects images to make some free space.
Same issue. Steps:
- Clone the repo
cd wazuh-docker/single-nodedocker-compose -f generate-indexer-certs.yml run --rm generator- Unlike @jay-oswald I imported the single-node docker-compose file into Synology's container manager software.
The error message is exactly the same as OP's.
So I found something I wouldn't call a fix, but a workaround. It looks like the issue is that the uid/gid the docker container runs as is problematic for Synology devices. My workaround is to chmod o+r the files in wazuh_dashboard and wazuh_indexer.
This is not a good idea, it's insecure. This is my first install of Wazuh so I don't know how critical these files are. I've not yet found any official documentation from Wazuh on the user/groups in their docker images, or the validity of changing them in docker-compose.
P.S. If you're like me and tried to stand it up a bunch of times before it worked, you may have to edit wazuh_dashboard/wazuh.yml to remove a bunch of duplicate entries.