magro
Support for AWS Tomcat 8 Stack 2.2
If you look here there is the documentation on the combination of apache, tomcat for the 2.2 stack. We are trying to upgrade as AWS is requiring this stack now but the sessions seem to be broken. Link to info
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts.platforms.html#concepts.platforms.java
Library info we are using in case that helps:
<artifactItem>
<groupId>de.javakaffee.msm</groupId>
<artifactId>memcached-session-manager</artifactId>
<artifactItem>
<groupId>de.javakaffee.msm</groupId>
<artifactId>memcached-session-manager-tc8</artifactId>
AWS SDK 1.1.29
Tomcat contextxml
<Manager className=\"de.javakaffee.web.msm.MemcachedBackupSessionManager\" memcachedNodes=\"${TOMCAT_SESSION_MEMCACHE_HOST}:11211\" sticky=\"false\" sessionBackupAsync=\"false\" requestUriIgnorePattern=\"^(GET /resources/.*|GET /security/healthcheck)$\" />
I still don't get the actual issue. What's the problem that you run into, how are your expectations not met?
This might help to understand what I'm asking for: http://sqa.stackexchange.com/questions/1920/best-guidelines-for-bug-reporting :-)
My apologies I guess I did not explain the root issue:
Issue Description: Two servers without sticky sessions are not sharing sessions in memcache. Sessions are not being saved to elasticache.
Step to Reproduce: Using elasticbeanstalk bring up environment using your memcache session manager and the Elasticache 2.2 stack version. The version for everything inside that was supplied above. Make sure there are two servers in the load balancer in order to show that sessions are not being saved to memcache and shared between the servers. Make sure sticky sessions is not on as the session will persist when you stay on one server, just not stored in memcache service. You will get an unknown session error on the server when you load balance to another server then the original that the session was created on.
Expected Results: Using Elasticbeanstalk stack version 2.1.1 it works fine and sessions are transfered between hosts behind the load balancer no problem. Instead of getting an unknown session when load balanced to another server you will simply continue in session as expected.
Actual Result: When you are load balanced to a new server it says "unknown session" in the error logs on the server and if it is a section they are not allowed they are blocked or redirected to the login page.
Awesome description! :-)
How's the loadbalancer configured, for sticky or non sticky sessions? Have you tried changing the msm configuration to sticky=true?
Do you see differences in the logs between elasticbeanstalk 2.1.1 and 2.2? Maybe you have to turn on debug logging (according to https://github.com/magro/memcached-session-manager/wiki/SetupAndConfiguration#configure-logging) to get more output.
The load balancer needs to be setup with non sticky sessions as it is our requirement.
I'll configure logging today and let you know.
I turned on sticky as it totally craps out without it. Here is the only log entries: 2016-09-19 17:49:04.589 INFO net.spy.memcached.ConfigurationPoller: Endpoint to use for configuration access in this poll NodeEndPoint - HostName:qa-pes-tomcat.enphsg.0003.use1.cache.amazonaws.com IpAddress:172.18.200.185 Port:11211
No error, but I wanted it always be storing the sessions in memcache but when I turn off sticky sessions no sessions seem to be working at all.
If this is all you get as log output you should check your log configuration.
If both the LB and msm are configured for non-sticky sessions it should work.
To me it's very hard to provide support based on the provided information, and also because I haven't used Beanstalk.
I'd suggest that you try to get a non-sticky setup running locally, with tomcat and your preferred lb. You can also check out the samples directory, perhaps it's useful to get started.
Yea thats my plan too. Had a bunch of other things to get too. Hopefully bring up a non-sticky today or tommorow and get you something more useful. Thanks.
I turned sticky off. But it doesn't seem to be an error. Same as before. But it does seem to throw the info log shown before every minute exactly. Perhaps I should turn the sessionBackupAsync="false" to true?
I'm somewhat out of ideas it's not logging any errors, but I can no longer login or keep a session accross multiple hosts.
Do you have it running locally now?
If you don't get any logs you have to fix this - seems to be a Beanstalk issue, not sure how I can help there.
I got this error:
WARNING [msm-storage-thread-4] de.javakaffee.web.msm.BackupSessionTask.call FAILED for session id 165590C8A3CA36174DD2BD9F3B1DAB6D java.lang.IllegalStateException: Client is not initialized
Got both errors fixed. Think the backupsession async might be reversed, but it's working now.
Great that you solved it. Can you share some insights, what caused the problem and what was the solution?
I am facing the same issue. Deamoner can you share some insights. Config and jars used. Thanks
I think I thought I fixed this issue by turning on the sticky=true and async_backup=true. This somewhat covers most cases as it moves the session into memcache eventually, but it appears on quick calls before the session is stored in the memcache by async that it doesn't work and is still broken. As a good example all our oauth calls seem to get broken as they create a new session that isn't in storage and quickly load balance back to bring back the token. So it appears the only broken portion is the session backup synchronously. Thoughts?
While the sticky setting changes the behavior fundamentally, sessionBackupAsync only changes the "guarantee" when the session is stored in memcached. I'd say you should use sessionBackupAsync="false".
I've tried that and the sessions do not work at all. The load balancer is not sticky and with no session backup async the sessions do not seem to be saving at all. The servers seem to be giving new sessions id constantly.
You could still try to get some more logs out of the system and share them, they might help to understand the issue.
The only time logs seem to appear is when I turn both those to true. But to give an example I have turned both options to false and then when I load balance between nodes no session is saved. The only log that occurs when it is these settings and why I think it's not working is the following log:
Catalina.out: 2016-10-05 13:26:07.569 INFO net.spy.memcached.ConfigurationPoller: Starting configuration poller. 2016-10-05 13:26:07.569 INFO net.spy.memcached.ConfigurationPoller: Endpoint to use for configuration access in this poll NodeEndPoint - HostName:staging-pes-sessions.apmlit.0003.use1.cache.amazonaws.com IpAddress:172.19.200.145 Port:11211 2016-10-05 13:27:07.568 INFO net.spy.memcached.ConfigurationPoller: Starting configuration poller. 2016-10-05 13:27:07.569 INFO net.spy.memcached.ConfigurationPoller: Endpoint to use for configuration access in this poll NodeEndPoint - HostName:staging-pes-sessions.apmlit.0001.use1.cache.amazonaws.com IpAddress:172.19.200.35 Port:11211
But if I turn them both to true then I see logs about creating and saving sessions, so it just appears that synchronous session saving is broken and never throws an error. I'd be happy to set you up with access to a system running as an example if you'd like.
This log output seems to create sessions but it never backs them up:
05-Oct-2016 13:32:58.912 FINE [http-nio-8080-exec-4] de.javakaffee.web.msm.MemcachedSessionService.createSession createSession invoked: null 05-Oct-2016 13:32:58.912 FINE [http-nio-8080-exec-9] de.javakaffee.web.msm.MemcachedSessionService.findSession Returning for session id 107A6D3E324693F4785660FC38A3BA9A (isConnectorSessionLookup: false, isIgnoredRequest: true) 05-Oct-2016 13:32:58.913 FINE [http-nio-8080-exec-13] de.javakaffee.web.msm.MemcachedSessionService.findSession Returning for session id 107A6D3E324693F4785660FC38A3BA9A (isConnectorSessionLookup: false, isIgnoredRequest: true) 05-Oct-2016 13:32:58.913 FINE [http-nio-8080-exec-9] de.javakaffee.web.msm.MemcachedSessionService.findSession Returning for session id 107A6D3E324693F4785660FC38A3BA9A (isConnectorSessionLookup: false, isIgnoredRequest: true) 05-Oct-2016 13:32:58.913 FINE [http-nio-8080-exec-4] de.javakaffee.web.msm.MemcachedSessionService.createSession Created new session with id 722CBE21B1B932282D406696809765F5 05-Oct-2016 13:32:58.913 FINE [http-nio-8080-exec-13] de.javakaffee.web.msm.MemcachedSessionService.findSession Returning for session id 107A6D3E324693F4785660FC38A3BA9A (isConnectorSessionLookup: false, isIgnoredRequest: true) 05-Oct-2016 13:32:58.913 FINE [http-nio-8080-exec-9] de.javakaffee.web.msm.MemcachedSessionService.findSession Returning for session id 107A6D3E324693F4785660FC38A3BA9A (isConnectorSessionLookup: false, isIgnoredRequest: true) 05-Oct-2016 13:32:58.913 FINE [http-nio-8080-exec-13] de.javakaffee.web.msm.MemcachedSessionService.findSession Returning for session id 107A6D3E324693F4785660FC38A3BA9A (isConnectorSessionLookup: false, isIgnoredRequest: true) 05-Oct-2016 13:32:58.914 FINE [http-nio-8080-exec-9] de.javakaffee.web.msm.MemcachedSessionService.findSession Returning for session id 107A6D3E324693F4785660FC38A3BA9A (isConnectorSessionLookup: false, isIgnoredRequest: true) 05-Oct-2016 13:32:58.914 FINE [http-nio-8080-exec-4] de.javakaffee.web.msm.RequestTrackingHostValve.invoke <<<<<< Ignored: GET /resources/themes/standard/images/icons/fugue/document-globe.png ================== 05-Oct-2016 13:32:58.914 FINE [http-nio-8080-exec-13] de.javakaffee.web.msm.MemcachedSessionService.findSession Returning for session id 107A6D3E324693F4785660FC38A3BA9A (isConnectorSessionLookup: false, isIgnoredRequest: true) 05-Oct-2016 13:32:58.914 FINE [http-nio-8080-exec-9] de.javakaffee.web.msm.MemcachedSessionService.findSession Returning for session id 107A6D3E324693F4785660FC38A3BA9A (isConnectorSessionLookup: false, isIgnoredRequest: true) 05-Oct-2016 13:32:58.915 FINE [http-nio-8080-exec-13] de.javakaffee.web.msm.MemcachedSessionService.createSession createSession invoked: null 05-Oct-2016 13:32:58.915 FINE [http-nio-8080-exec-9] de.javakaffee.web.msm.MemcachedSessionService.createSession createSession invoked: null 05-Oct-2016 13:32:58.915 FINE [http-nio-8080-exec-9] de.javakaffee.web.msm.MemcachedSessionService.createSession Created new session with id 71E83A4C04A98AFFBDCDF3CACE0451BD 05-Oct-2016 13:32:58.915 FINE [http-nio-8080-exec-13] de.javakaffee.web.msm.MemcachedSessionService.createSession Created new session with id 1DB11737620C77561C72CF01C0A4AD9B 05-Oct-2016 13:32:58.916 FINE [http-nio-8080-exec-9] de.javakaffee.web.msm.RequestTrackingHostValve.invoke <<<<<< Ignored: GET /resources/themes/standard/images/menu-border.png ==================
So if I just turn off sticky="false" and async="false" there are no errors and no logs but the polling for configuration and I cannot even login one server. So It appears non-sticky on our environment doesn't work at all and no logs appear. If I turn on sticky="true" then there are logs and I am able to login one server.
Honestly, I'm running out of ideas how to help you.
Because the setup seems to be quite similar as in #314 perhaps @rajeshkibo could step in and give some hints on how to get this setup to work?
+1 on v2.2 memcached issue, using it for tomcat8 session caching. No exceptions in the logs, but session isn't found right after it was cached: 2016-10-08 02:04:13,726 HttpSession being created as SecurityContext 2016-10-08 02:04:13,927 No HttpSession currently exists
This is after I changed from v2.0.1 to v2.2, when downgraded to v2.1.1 everything started to work again, you guys should take a close look at what is it you changed in v2.2.
@Bobik007 Not sure if I understand you correctly.
v2.2 is Beanstalk, therefore it's not the question what changed in msm but what changed in Beanstalk so that it no longer works with msm (or msm no longer works with Beanstalk).
As already said I myself haven't ever used Beanstalk therefore I cannot provide Beanstalk specific help. I also don't have the time to setup Beanstalk with msm to analyze this issue for you.
It would be great if anybody of you guys could find the root cause and a solution.
My bad, did not realize you were looking for a problem in memcached, this is definitely beanstalk issue and should be taken care of by AWS folks. Sorry for confusion.
Why would AWS folks take care of this, they don't support msm, do they? If Beanstalk 2.2 has issues with logging or anything in the standard/supported setup they'd have to investigate / fix it, otherwise not. So the question remains what's causing the issue. Have you tried to get msm running using the basic building blocks used by Beanstalk (tomcat etc) without using Beanstalk, in an environment you can control?