Splunk kubernetes (EKS) cluster getting constant "Your session has expired. Log in to return to the system." error

[blakecschmidt]

trafficstars

This issue I am seeing is identical to this issue that was posted a while ago by someone else but I don't believe the reason behind their problem applies to my situation so I am asking again.

What I am doing is I am creating a Kubernetes (EKS) cluster in AWS to run Splunk on and just to get a barebones example of it running I have been following this tutorial pretty closely. The main difference between my implementation and the one shown in the tutorial is I have a very simple load balancer service set up in the cluster which has an external IP I can access the Splunk web UI from.

When I access the Splunk web UI, I log in with the default credentials and it then gives me the error "Your session has expired. Log in to return to the system." I tried to clear my browser cache, tried a different browser, had a coworker try on his machine, and got off my company's VPN and none of these things resolved the issue.

From the few resources I have seen online about this error, it seems like there was some issue where this error occurs if there are more than one docker containers running on the same host which breaks the session-key cookie and gives this error. To my knowledge there is only one docker image running and I even reduced the instances in my cluster from 2 down to 1 and still have this issue.

Any guidance on this problem would be really helpful and I can provide as much information as needed on my setup. I am pretty new to Splunk in general and on top of that there is very little information online when it comes to using Kubernetes with Splunk so I'm hoping this is a good place to ask these questions.

[jnichols3]

As far as I know, Splunk web still cannot run behind an AWS ELB without significant side car work (like a reverse proxy). This is the case in EC2 and the same problems manifest themselves in EKS. There are various posts in splunk answers on this: https://community.splunk.com/t5/Security/How-to-configure-Splunk-Enterprise-in-front-of-AWS-ELB-instance/m-p/502748

You can use an nginix container and it sort of works, but I usually just end up using DNS with an A record for multiple splunk web instances.

As far as guidance is concerned, that tutorial is really old at this point and I'd encourage you to use the Splunk Operator- https://www.splunk.com/en_us/blog/platform/going-live-splunk-operator-for-kubernetes-1-0-0.html - that will give you multiple search heads in a cluster.

That being said that tutorial still looks like it would work just fine if you can't use operators yet.