Memory leak in Undertow (2.15.1)

[vellrya]

Hello.

For a long time I have been trying to detect a memory leak in my program, but after tests it turns out that the leak is not in my code. I built the project with Jooby 2.13.0 and Kotlin 1.4.32 - no leaks. Then I built the project with 2.15.1 and Kotlin 1.6 without any code changes and found memory leaks again. I'm attaching a screenshot of heap analysis in MAT, maybe I should clarify some details from MAT (I haven't dealt with memory leaks before) - I'd be happy to share it, or maybe I should report it to Undertow?

I would like to note that the leak occurs in project with active SSE usage, in my other projects (without SSE) I haven't encountered OOM even on 2.15.1 version, but I haven't investigated their memory consumption carefully either.

[vellrya]

I manage sse connections as below (simplified):

val paidSSE = ConcurrentHashMap<String, ServerSentEmitter>()

//...

sse("/path") {
  sse.onClose {
    for (tmpSse in paidSSE) {
      if (tmpSse.value==sse) {
        paidSSE.remove(tmpSse.key)
        break
      }
    }
  }
  sse.keepAlive(14000)
  paidSSE[ctx.header("id").value()] = sse
  return@sse ctx
}

Probably you know better way. but this works without issue on 2.13.0)

[vellrya]

Hello, I am still using version 2.13.0 as it has no memory leaks. Should I try version 3.x, or has there been no change related to my problem?

[jknack]

Hi, nothing changed in Jooby and I couldn't find any bug related in Undertow.

Jooby 3.x has a new version of undertow you can try that or in 2.x you can override the undertow dependencies and try that too.

[vellrya]

Thanks, as soon as 3.0.0.M3 with Kotlin 1.8.0 appears, I will try the new version)

[jknack]

jooby 2.13.0 was released with undertow 2.2.14.Final

while jooby 2.16.1 (latest 2.x) uses: undertow 2.2.19.Final and xnio 3.8.7.Final

In your project try this:

• upgrade jooby to latest 2.16.1
• then on your pom do:

<!-- https://mvnrepository.com/artifact/io.undertow/undertow-core -->
<dependency>
    <groupId>io.undertow</groupId>
    <artifactId>undertow-core</artifactId>
    <version>2.2.14.Final</version>
</dependency>

<dependency>
    <groupId>io.jooby</groupId>
    <artifactId>jooby-utow</artifactId>
    <version>2.16.1</version>
    <exclusions>
       <!-- https://mvnrepository.com/artifact/org.jboss.xnio/xnio-api -->
         <exclusion>
              <groupId>org.jboss.xnio</groupId>
              <artifactId>xnio-api</artifactId>
         </exclusion>
    </exclusions>
</dependency>

[vellrya]

This is the dependency tree with Kotlin 1.6.21, I'll report on the result.

[vellrya]

I tried it first with the configuration you suggested and found no memory leaks. Then removed all the exclusions and switched completely to version 2.16.1 and in 64 hours of use there is no sign of any memory leaks. Hopefully the problem has been resolved, I'm closing the issue)

UPD: Can you please take a look at https://github.com/jooby-project/jooby/issues/2462

[vellrya]

Unfortuanetely, the issue has not been fixed. This is my mistake - I changed the dependencies in the .pom file, but forgot to change the output artifacts. In fact, there were no changes and I continued to use version 2.13.0 with Kotlin 1.4.32, where there really is no memory leak.

I've updated the Jooby version in the output artifact and unfortunately the problem is relevant to version 2.16.1.

[2023-01-25 15:44:27,726]-[main] INFO io.undertow - starting server: Undertow - 2.2.19.Final [2023-01-25 15:44:27,733]-[main] INFO org.xnio - XNIO version 3.8.7.Final [2023-01-25 15:44:27,740]-[main] INFO org.xnio.nio - XNIO NIO Implementation Version 3.8.7.Final [2023-01-25 15:44:27,831]-[main] INFO org.jboss.threads - JBoss Threads version 2.3.6.Final

In the pictures below you can see that memory usage is constantly increasing, although on 2.13.0 memory usage was in the same range and did not increase.

I captured two heap snapshot with 10 min difference and here the result of dump comparison:

Leak suspect 1 in MAT:

Leak suspect 2 in MAT:

I'll take your recommendation (https://github.com/jooby-project/jooby/issues/2607#issuecomment-1377134016) and check for memory leaks with the correct version of the undertow and xnio dependencies in the output artifact.

[vellrya]

So, here are the results using the dependency versions you specified in https://github.com/jooby-project/jooby/issues/2607#issuecomment-1377134016:

[2023-01-25 19:06:23,821]-[main] INFO io.undertow - starting server: Undertow - 2.2.14.Final [2023-01-25 19:06:23,828]-[main] INFO org.xnio - XNIO version 3.8.4.Final [2023-01-25 19:06:23,836]-[main] INFO org.xnio.nio - XNIO NIO Implementation Version 3.8.4.Final [2023-01-25 19:06:23,942]-[main] INFO org.jboss.threads - JBoss Threads version 3.1.0.Final

(Jooby 2.16.1)

From 51 MB heap to 230 MB (after forcing GC) in five hours.

[vellrya]

Test on Jooby 2.13.0, no memory leaks. No changes made to code - just different versions of dependencies.

[vellrya]

I was curious to find out in which version the memory leak appeared in order to localise the problem. Jooby 2.14.0 .. 2.14.2 - no leak:

[2023-01-26 11:55:34,780]-[main] INFO io.undertow - starting server: Undertow - 2.2.17.Final [2023-01-26 11:55:34,788]-[main] INFO org.xnio - XNIO version 3.8.7.Final [2023-01-26 11:55:34,797]-[main] INFO org.xnio.nio - XNIO NIO Implementation Version 3.8.6.Final [2023-01-26 11:55:34,875]-[main] INFO org.jboss.threads - JBoss Threads version 2.3.6.Final

Jooby 2.15.0 - leak:

[2023-01-26 16:45:13,888]-[main] INFO io.undertow - starting server: Undertow - 2.2.17.Final [2023-01-26 16:45:13,895]-[main] INFO org.xnio - XNIO version 3.8.7.Final [2023-01-26 16:45:13,913]-[main] INFO org.xnio.nio - XNIO NIO Implementation Version 3.8.6.Final [2023-01-26 16:45:13,936]-[main] INFO org.jboss.threads - JBoss Threads version 2.3.6.Final

Note that dependencies have not changed.

I'm not very good at using MAT, but perhaps this will give you an idea:

Option "Merge shortest paths to GC Roots" on io.undertow.server.protocol.http.HttpOpenListener$1 objects on 2.14.2 heap dump gave the following results: Where 300+18 - is a real number of opened connection.

Output for 2.15.0:

Also io.undertow.server.HttpServerExchange objects: 2.14.0

2.15.0

Huge number of references in 2.15.0 from sun.nio.ch.WEPollSelectorImpl, and the number keeps increasing. Since the dependencies haven't changed, is it possible that the problem is inside the JDK and related to the switch to JDK 17? This commit was made between 2.14.2 and 2.15.0: https://github.com/jooby-project/jooby/commit/f25d3aebde3a054bd32dc997b9eaaf7fd0751f58 For reference, I use JDK 17 to build and run the project.

[vellrya]

I should have tried 3.0.0.M2 from the start :)

Looks great.