OOM after migrating to 3.2.6

[ennishol]

Bug description

After migrating to the latest camel-quarkus, I get OOM. The route looks similar to the following

from("file://csv")
  .split(body())
  .to("seda:line")
from("seda:line")
   .setProperty(LINE, body())
   .setProperty(ID, simple("${body.id}"))
   .setHeader(Exchange.BREADCRUMB_ID, simple("id-${body.id}")))
      .... do work ...
   .to("seda:another")
from("seda:another")
   .setHeader(Exchange.BREADCRUMB_ID, simple("id-${body.id}")))
      .... do work ...

I captured the histogram

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:       4069129      240857960  [Ljava.lang.Object; ([email protected])
   2:       1737404      152891552  java.lang.reflect.Method ([email protected])
   3:       5170893      124101432  java.util.ArrayList ([email protected])
   4:       3522175      112709600  java.util.HashMap$Node ([email protected])
   5:       1713630       95963280  org.apache.camel.component.bean.MethodInfo
   6:       1713630       41127120  org.apache.camel.component.bean.MethodInfo$ParameterExpression
   7:         65063       38403392  [Ljava.util.HashMap$Node; ([email protected])
   8:       1141384       36524288  org.apache.camel.component.bean.ParameterInfo
   9:       1713630       36517232  [Lorg.apache.camel.Expression;
  10:       1137394       27297456  org.apache.camel.support.builder.ExpressionBuilder$35
  11:       1141405       18262600  [Ljava.lang.annotation.Annotation; ([email protected])
  12:        378378       17844040  [B ([email protected])
  13:        480239       15367648  java.util.concurrent.ConcurrentHashMap$Node ([email protected])
  14:         35202       10113096  [Ljava.util.concurrent.ConcurrentHashMap$Node; ([email protected])
  15:        227605        9104200  java.util.LinkedHashMap$Entry ([email protected])
  16:        372213        8933112  java.lang.String ([email protected])
  17:        143920        8059520  java.util.LinkedHashMap ([email protected])
  18:        160262        3846288  java.lang.Long ([email protected])
  19:         67231        2689240  java.util.EnumMap ([email protected])
  20:         33443        2407896  org.apache.camel.support.DefaultExchange
  21:         35847        2294208  java.util.concurrent.ConcurrentHashMap ([email protected])
  22:        142868        2285888  java.util.LinkedHashMap$LinkedEntrySet ([email protected])
  23:         33443        2140352  org.apache.camel.support.ExtendedExchangeExtension
  24:         50586        2023440  java.math.BigDecimal ([email protected])
  25:         15741        1853016  java.lang.Class ([email protected])
  26:         69119        1658856  com.me.MyId

1713630 instances of org.apache.camel.Expression looked suspicios to me, so I replaced all expresions in the route with processors

from("seda:line")
      .process(e -> {
                e.setProperty(LINE, e.getIn().getBody())
                e.setProperty(ID, e.getIn().getBody(MyBean.class).getId())
                e.getIn().setHeader((Exchange.BREADCRUMB_ID,  e.getIn().getBody(MyBean.class).getId())
     })
           .... do work ...

and this solved the problem. No OOM anymore. The number of Expression remains constant by 2000 instances only during the execution

Unfortunately, I can't share the code, but maybe it is worth checking what could be the reason for the large number of Expression instances

[ppalaga]

@davsclaus @oscerd is this by any chance a known issue in Camel 4.0.0?

[davsclaus]

Upgrading from what to what ?

[ennishol]

Upgrading from camel quarkus 2.16.x to 3.2.6. It would mean from camel 3 to 4 I believe

[ppalaga]

Upgrading from camel quarkus 2.16.x to 3.2.6.

Those sound like Quarkus Platform versions. The underlying CQ in Platform 3.2.6 would be 3.2.0 (Camel 4.0.0)

[ennishol]

https://camel.apache.org/blog/2023/09/camel-quarkus-release-3.2.0/

" Camel Quarkus 3.2.0 is based on Quarkus 3.2.5.Final. However, it’s important to note that this version is available in Quarkus platform 3.2.6.Final onward. Therefore, users are encouraged to upgrade to Quarkus Platform 3.2.6.Final for the latest features and improvements. "

[oscerd]

Camel-Quarkus is a thing, Quarkus-platform a different one.

[ennishol]

I've done allocation profiling with the async profiler. Here is the flame graph:

java.lang.reflect.Method 309.311 Samples, 54%

[davsclaus]

you can try in pure camel 4.0 without quarkus, and also try when 4.0.1 is released, or a 4.1.0-SNAPSHOT

[ennishol]

@davsclaus The application depends "heavily" on Quarkus cdi, config etc. I''m not sure how I would just run it without quarkus. I'd have to rewrite everything. Maybe I just wait for a camel-quarkus SNAPSHOT version with newer camel

[davsclaus]

yeah 4.0.1 is to be released in a week or so (if all goes well)

[davsclaus]

Related to https://issues.apache.org/jira/projects/CAMEL/issues/CAMEL-20035