Huge performance drop when migrating from 6.4.0 to 6.6.0

[TomasTokaMrazek]

Description: Mediations are 10 - 20 times slower on 6.6.0 than on 6.4.0. For example símple payloadFactory with two arguments can take up to 100ms instead of 1ms. I used exactly the same CAR, both EI are on the same server.

synapse.properties, carbon.xml, axis2.xml, integrator.sh are configured in the same way (for example Xms and Xmx memory allocation).

Disabling XPath 2.0 fallback doesn't change anything.

6.4.0

[2020-04-15 07:02:12,661] [-1234] [] [PassThroughMessageProcessor-93]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  SDX-EA-TokenVerifySequence = 1
[2020-04-15 07:02:12,671] [-1234] [] [PassThroughMessageProcessor-93]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  SDX-EA-TokenVerifySequence = 2
[2020-04-15 07:702:12,671] [-1234] [] [PassThroughMessageProcessor-93]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  SDX-EA-TokenVerifySequence = 3
[2020-04-15 07:02:12,671] [-1234] [] [PassThroughMessageProcessor-93]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  SDX-EA-TokenVerifySequence = 4
[2020-04-15 07:02:12,672] [-1234] [] [PassThroughMessageProcessor-93]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  SDX-EA-TokenVerifySequence = 5

6.6.0

TID: [-1234] [] [2020-04-15 09:10:41,785]  INFO {org.apache.synapse.mediators.builtin.LogMediator} - SDX-EA-TokenVerifySequence = 1
TID: [-1234] [] [2020-04-15 09:10:41,788]  INFO {org.apache.synapse.mediators.builtin.LogMediator} - SDX-EA-TokenVerifySequence = 2
TID: [-1234] [] [2020-04-15 09:10:41,789]  INFO {org.apache.synapse.mediators.builtin.LogMediator} - SDX-EA-TokenVerifySequence = 3
TID: [-1234] [] [2020-04-15 09:10:41,789]  INFO {org.apache.synapse.mediators.builtin.LogMediator} - SDX-EA-TokenVerifySequence = 4
TID: [-1234] [] [2020-04-15 09:10:41,859]  INFO {org.apache.synapse.mediators.builtin.LogMediator} - SDX-EA-TokenVerifySequence = 5

Affected Product Version: 6.6.0

OS, DB, other environment details and versions:
Tested on two environments

Ubuntu 18.04 6.4.0 - OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_192-b12) 6.6.0 - Java(TM) SE Runtime Environment (build 1.8.0_211-b12)

CentOS 7 6.4.0 - Java(TM) SE Runtime Environment (build 1.8.0_192-ea-b02) 6.6.0 - OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.5+10)

[mfkasim]

Hello TomasTokaMrazek, We are facing a similar issue while doing our load tests which indicated a huge difference between json and xml as a payload. While json get around 900 TPS, XML stuck at around 70 TPS. We still cannot figure out what the actual problem is.

If you could solve this issue, please let us know the way. also, if possible send the your configs so that we can check the status.

[mfkasim]

Hello TomasTokaMrazek, We have done further tests regarding 6.6 and 6.5 (which acts similarly to 6.4 in performance) and still see the samee bahavior. It seems that the root cause to the performance issue is related to Dynamic class loading for XML DocumentBuilderFactory and DomParser Configuration object.

To overcome this unreasonable performance overhead, we updated the JVM parameters to directly refer to the two classes needed directly to simplify XML classes loading. This return us to a comparable performance regarding 6.6 vs. 6.5 or 6.6 vs. 6.4

I hope this help you in return and I hope if the WSO2 can describe or examine why this behavior is happening.

-Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl \
-Dorg.apache.xerces.xni.parser.XMLParserConfiguration=org.apache.xerces.parsers.XIncludeAwareParserConfiguration \

[TomasTokaMrazek]

Last year I've spend about two weeks trying to fix this, we eventually had to rollback to 6.4.0 and there was no further time allocation to burn on this.

Lately I've been successfully experimenting with breaking up WSO2 apart and replacing old XML parsers. We are currently using Saxon 10.3, which gives us the ability to use XSLT 3.0 in xslt mediator and completely ignore Xalan. I was hoping to replace Jaxen, but that is unfortunately baked in SynapsePath. There I discovered, how Xpath 2.0 fallback actually works and learned, that there is an overhead. If Xpath 1.0 function fails via Jaxen, it falls back to other implementation. Jaxen is called for every xpath evaluation.

Anyway, I wanted to force JAXP factories to mentioned Saxon library and I wanted to try exactly your solution - well instead of using Xerces. I'll report the results once I've got the time to actually test it. Thank you for validating the idea, now I have a bit of hope to not waste my time. Using 6.4.0 with no support for JDK 11 is a bit problematic in a long run.

[TomasTokaMrazek]

@mfkasim So I finally have time and allocation to test it. It helped quite a lot in my envrionment, but it's still 2 times slower than 6.4.0. It is better than 10 times slower without the properties, but I am afraid it's not good enough to make the switch in production.

[alimalek71]

Hi guys,

Can I ask you to share your load test results? and explain what was your test env configs? (like CPU, Ram, Server Tunning Config, Application Tunning Config)

I want to measure my app performance and your result can help me. I want to determine EI performance and how much TPS the product can respond by calling just EI health check api and after that proxying my application through EI with high timeout?

Thanks in advance

@TomasTokaMrazek @mfkasim