Session metrics are decreasing

[sgflt]

Description

---

Metrics with following definition are decreasing in one module.

<metadata>
      <tags>
        <tag>
          <name>module</name>
          <value>%s</value>
        </tag>
      </tags>
      <name>session.module.active.count</name>
      <mbean>amx:pp=/mon/server-mon[$SERVER_NAME],type=session-mon,name=cis-web-ui/%s/server/activesessionscurrent#current</mbean>
      <type>gauge</type>
      <unit>none</unit>
      <displayName>Count of active sessions</displayName>
      <description>The total of active sessions</description>
    </metadata>

Expected Outcome

Gauge to be only positive.

Current Outcome

Steps to reproduce (Only for bug reports)

Samples

Context (Optional)

Environment

• Payara Version: 5.2020.7
• Edition: Full
• JDK Version: OpenJDK 11
• Operating System: RedHat

[AlanRoth]

Hi @sgflt, can you provide a more clear reproducer? Is this just in the default payara metrics application? Thank you.

[sgflt]

Hi @AlanRoth , I need to collect more data. Yes, it is default Payara's implementation configured via metrics.xml.

[sgflt]

Found hint in code:

// PAYARA-486 & PAYARA-1224 Correct the incorrect incrementation - needs further looking at

Incomplete attempts:

https://github.com/payara/Payara/pull/561 https://github.com/payara/Payara/pull/1217

[sgflt]

Method sessionDestroyedEvent is caled at least 20 times when hitting F5 on specific page. Session id stays the same.

[sgflt]

FlashLightProbe#invokerList contains 36 identical listeners.

{Integer@15119} 142 -> {ReflectiveClientInvoker@15120} "id=142, target=org.glassfish.web.admin.monitor.SessionStatsProvider@2ea1e2fe, method=public void org.glassfish.web.admin.monitor.SessionStatsProvider.sessionDestroyedEvent(java.lang.String,java.lang.String,java.lang.String), paramNames=[sessionId, appName, hostName], probeIndices=[0, 1, 2], useProbeArgs=true, hasComputedParams=false"
{Integer@15111} 158 -> {ReflectiveClientInvoker@15096} "id=158, target=org.glassfish.web.admin.monitor.SessionStatsProvider@31998dfa, method=public void org.glassfish.web.admin.monitor.SessionStatsProvider.sessionDestroyedEvent(java.lang.String,java.lang.String,java.lang.String), paramNames=[sessionId, appName, hostName], probeIndices=[0, 1, 2], useProbeArgs=true, hasComputedParams=false"

[sgflt]

It seems that each war in ear registers own listener hook, but event does not care about war locality and executes all attached listeners from whole ear.

[sgflt]

Fix seems to be easy... crafting a new code.

[AlanRoth]

Hi @sgflt, thanks for keeping me updated, feel free to make a PR when you are ready. Thank you.

[sgflt]

And fix is already present as isValidEvent method, and I am looking at a wrong place.

[sgflt]

I have Examined positive values and they do not seem to be correct too. Some event happened and all numbers are going mad.

[AlanRoth]

Hi @sgflt, could you provide a sample app so I can quickly verify this issue? Thank you.

[sgflt]

Demo app located at https://github.com/sgflt/payara-test-case should be sufficient. I though I knew wehere the problem is, but after few hours I got nothing. The cause is well hidden and I dont know what order of steps can cause this behavior. I plan to add metrics for pasivation and activation of sessions in hope it might be related.

[AlanRoth]

Hi @sgflt, I'm not sure how to use your Demo app to reproduce your issue. Could you provide some steps?

[sgflt]

Hi @AlanRoth steps for deplyoment and basic clicking are reusable from https://github.com/payara/Payara/issues/3907. I will prepare configuration of metrics.xml to my repo.

[sgflt]

metrics.xml pushed right now

[sgflt]

New data afted added pasivated and activated metrics.

[AlanRoth]

Hi @sgflt,

I apologise about the hiatus.

Using the metrics.xml in the payara-test-case reproducer, I'm not able to see the vendor_session_module_* statistics at the metrics endpoint. All other vendor metrics show up and look normal.

Is there any further configuration required to view the session module metrics?

Are you still experiencing this issue with the latest Payara version?

Thanks, Alan

[sgflt]

Hi @AlanRoth ,

metrcis.xml is correct and it should produce session metrics. Do you have HIGH level in monitoring configuration set up on all properties? (I am not sure if Web Container is the only component, what we are interested in)

Configuration of MBeans can be verified through JVisalvm MBean plugin, that displays all enabled and loaded MBeans.

The latest Payara still produces inaccurate values, but not negative ones (it may be just coincidence). There is non zero bottom line, for example count of active sessions in one module bounces between 1200 and 800. Count never goes close to zero, which is not expected.

[sgflt]

Alternatively it may reveal another problem with SSO. Sometimes users experience no log out even if all sessions should be expired after configured session timeout. Some sessions are semi accessible even after expiration.

[AlanRoth]

Hi @sgflt,

I must apologise as this issue has fallen through the cracks and has not recieved any attention.

I will close this issue at the moment, however I encourage you - if the issue is still relevant - to raise a new issue and a member of our team will handle it.

Thank you, Alan