forwarder latest docker image fails to start

[alexeadem]

Image: splunk/universalforwarder:latest

You can reproduce it with:

https://github.com/splunk/docker-splunk/blob/develop/test_scenarios/1uf1so.yaml

or as described here:

https://splunk.github.io/docker-splunk/EXAMPLES.html#create-standalone-and-universal-forwarder

Universal forwarder never starts with the the following error:

[WARNING]: Using world-readable permissions for temporary files Ansible needs to create when becoming an unprivileged user. This may be insecure. For information on securing this, see
https://docs.ansible.com/ansible/user_guide/become.html#risks-of-becoming-an-unprivileged-user
^C [ERROR]: User interrupted execution
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
Error calling execve(): No such file or directory
Error launching  command: No such file or directory
splunkd is not running.

=> Working fine in 8.2.9.

[anthonybrochu]

I'm also having the same problem, the splunkd never launches, except when reverting to version 8.2.9

[Adam2Marsh]

I've raised this to splunk via there support portal; case ref is 3200123

Will update this once I hear back.

[Adam2Marsh]

This looks to be a duplicate of https://github.com/splunk/docker-splunk/issues/557

[Adam2Marsh]

I got the following response; it's a known issue

---

This is a known Docker bug, its currently working in progress with Expected fix: 9.1.0 As of now the workaround is as follows

Workaround:

• Remove: /opt/splunkforwarder/etc/myinstall/splunkd.xml file then start Splunk

Or this can be confgured:

• Add the following to the compose file: splunk: tty: true

https://community.splunk.com/t5/Installation/Upgrading-Universal-Forwarder-8-x-x-to-9-x-x-does-not-work/m-p/619919_

I've asked for a date when version 9.1.0 will be released.

[Adam2Marsh]

This was fixed in 9.0.5; I haven't tested it myself but assured it is.

[flaken2000]

Found this issue today because I've been having this exact issue with version 9.0.5. So it is not fixed with 9.0.5. Looks like will need to stick with 8.2X until version 9.1.0 is released with a fix for this issue.

[gergelydavidwork]

Same here @flaken2000 , it is NOT fixed with 9.0.5! :(

[tnedeczki]

I can't believe why it's taking so much time to fix this bug, which has been going on since September (https://github.com/splunk/docker-splunk/issues/557). Instead, you have to use 8.2.X versions which are quite risky from a security point of view.

The whole ansible process gets stuck when it comes to a task where it needs to call ./splunk.

For example:

• get_splunk_status task: https://github.com/splunk/splunk-ansible/blob/develop/roles/splunk_common/tasks/get_splunk_status.yml#L10
• if the following SPLUNK_BEFORE_START_CMD flag is used, this task will also be stuck: pre_splunk_start_commands https://github.com/splunk/splunk-ansible/blob/develop/roles/splunk_common/tasks/pre_splunk_start_commands.yml

If you go inside the container and kill of those process, you will get the following error message:

Error calling execve(): No such file or directory
Error launching  command: Invalid argument

[gergelydavidwork]

I can't believe why it's taking so much time to fix this bug, which has been going on since September (#557). Instead, you have to use 8.2.X versions which are quite risky from a security point of view.

The whole ansible process gets stuck when it comes to a task where it needs to call ./splunk.

For example:

• get_splunk_status task: https://github.com/splunk/splunk-ansible/blob/develop/roles/splunk_common/tasks/get_splunk_status.yml#L10
• if the following SPLUNK_BEFORE_START_CMD flag is used, this task will also be stuck: pre_splunk_start_commands https://github.com/splunk/splunk-ansible/blob/develop/roles/splunk_common/tasks/pre_splunk_start_commands.yml

If you go inside the container and kill of those process, you will get the following error message:

Error calling execve(): No such file or directory
Error launching  command: Invalid argument

It's been almost 1 year since it was broken :(

[Adam2Marsh]

Hey All,

Re-raising back to splunk now... as I've doubled check and yep you still need the workaround to make 9.0.5

[adityapinglesf]

9.0.5 was updated yesterday. I was able to get the UF running with the following command -

 docker run -p 8000:8000 -p 8089:8089 -e "SPLUNK_PASSWORD=trypass@" -e "SPLUNK_START_ARGS=--accept-license" --name uf-1 -it splunk/universalforwarder:9.0.5

Can you try now with the newly released image?

[tnedeczki]

Hi @adityapinglesf

Thanks, it is working now.

[flaken2000]

Confirmed! Working for us too.

[shankar1294]

Hi, I am facing the same above error in 9.x versions. I am using this linux command to start the splunk server. /opt/splunkforwarder/bin/splunk start --accept-license This command works with lower versions. Can anyone pls help?