Splunk on kubernetes

[xeor]

I have been fighting very weird errors trying to get splunk working inside a kubernetes cluster.

Setup:

• Storage: nfs
• Image-version: 6.6.2
• kubernetes: 1.7.1 on Ubuntu 16.04.2 LTS

Mounting /opt/splunk/etc and /opt/splunk/var always gave me errors like https://answers.splunk.com/answers/312247/after-upgrading-a-search-head-cluster-to-splunk-63-1.html. Some of the resources it tried to get was also showing __raw/..../undefined/... where the undefined part was ment to show the username (admin). Lots of small things didn't work.

After a lot of trial and errors, I got it to work with mounting in separate directories under /opt/splunk/var, like spool and run.. It was a lot of trial and errors. But now, splunk gave me errors like ERROR while running renew-certs migration. and Warning: cannot create "/opt/splunk/var/run/splunk" when kubernetes recreated it.

What seams to work is this:

• Run with SPLUNK_USER set to root
• Mount in /opt/splunk/etc, /opt/splunk/var/lib, and /opt/splunk/var/log on their own.
  • /opt/splunk/var/log for good measure...
  • Mounting /opt/splunk/var will give the errors above, even if run as root.

[outcoldman]

We just published first version of our application "Monitoring Kubernetes" https://splunkbase.splunk.com/app/3743/ and collector https://www.outcoldsolutions.com. Please take a look on our manual how to get started https://www.outcoldsolutions.com/docs/monitoring-kubernetes/

[halr9000]

Way to go, @outcoldman! Tried it with OpenShift by chance? I'll have to spin up a lab.

[outcoldman]

@halr9000 I have not tried it yet in OpenShift! Let me know if you will get it to work.