Bug Report: Memory leak when using @Dependent instances

[bvschaik]

Brief Summary

Payara 6.2025.1 and up contain a memory leak in handling objects with a @Dependent annotation.

In any class, inject a dependency annotated with @Dependent and manually get and destroy an instance:

@ApplicationScoped
class SomeController {
    @Inject
    private Instance<SingleUseDependentClass> singleUseDependentClassInstance;

    public void doSomething() {
        var instance = singleUseDependentClassInstance.get();
        instance.doComplexOperation();
        singleUseDependentClassInstance.destroy(instance);
    }
}

Upon calling destroy(), the @PreDestroy method on the instance is called, but the instance itself is not destroyed, leading to a memory leak and a crash of the server once enough objects are retained to cause an OutOfMemoryError.

Expected Outcome

Created objects are disposed of properly, memory usage remains constant, server should stay up and running.

Current Outcome

Created objects are retained indefinitely, causing rising memory usage, eventually resulting in an OutOfMemoryError.

Reproducer

See: https://github.com/bvschaik/payara-dependent-reproducer

1. Build a .war from above project (mvn package) and deploy it on Payara
2. Do a GET to /api/memory a few times and note the memory usage going up by 100MB every time
3. Note in the logs that the finalize method is never called
4. Depending on the available memory to the server, an OOM error will happen within a limited number of requests

Operating System

MacOS, Linux

JDK Version

17

Payara Distribution

Payara Server Web Profile

[rignicat]

@Pandrex247 @lprimak We expected this to be resolved in 6.2025.3 with the revert #7245 but this doesn't appear to be fixed yet

[lprimak]

I don't think this has anything to do with what I am working on.

[marcreichman-pfi]

This may be connected with #7170 - it also showed up in 6.2025.1 (before the CDI work from @lprimak i believe?). I also observed leaks around the same time but have not had time to dig in and investigate. Our app uses MANY short lived objects using this pattern of instance .get() and .destroy(object) and has for many years. Suddenly the heap usage was skyrocketing. I hope with @bvschaik 's reproducer this can be figured out so we're not stuck on 6.2024.x and earlier builds indefinitely.

Thanks @bvschaik !

[marcreichman-pfi]

It looks like WELD changes went into 6.2025.1 (jan 9) to try to fix TCK, and it's a patched version of weld 5.0.1, even though weld for EE 10 is on 5.1.x and EE 11 6.x is out. One of the other weld updates since in 5.1.x mentions a leak cleanup. I suspect this and this Payara change introduces this leak but I'm not qualified enough to spot it in the weld changes since there are quite a few changes.

I'm hoping someone can take a look at this from the perspective of the weld changes and this reproducer to come up with an idea of why objects are not being collected suddenly.

[bvschaik]

I created a heap dump and ran it through the Eclipse Memory Analyzer. It looks like there's some recursion/circular reference going on. This screenshot is a trace of inbound references from one of the instances; you can keep expanding the "parentCreationalContext" objects:

[marcreichman-pfi]

@bvschaik That is consistent with the class names related to some of those changes above. So it seems like the updated / patched version of weld 5.0.x is exhibiting this issue. Hopefully this is enough useful information for either a fix or a revert of the weld version integration!

[marcreichman-pfi]

@bvschaik @artur-mal some more evidence: https://github.com/payara/Payara/issues/7170#issuecomment-2768536284

[marcreichman-pfi]

@artur-mal Is there anyone who might be able to look into this? This is kind of keeping us dead in the water on 6.2024.12.

[artur-mal]

@marcreichman-pfi, let me test the latest Payara Community Server release using the provided reproducer.

[artur-mal]

I've successfully reproduced the issue and have escalated it to the platform development team under the reference FISH-10947.

[marcreichman-pfi]

Hi @artur-mal ,

Just curious if anyone has gotten eyeballs on this yet? I realize it's community support but this is keeping us (and I suppose a number of other users) on 6.2024.12 and earlier.

[marcreichman-pfi]

Hi @artur-mal,

Respectfully would it be possible to get an update on this one? There is no upgrade path forward, community or otherwise, past 2024.12 with this issue. I suspect I may need to build a forked copy with the older version of WELD which was used. The TCK is important but a stable service is more so. It seems like weld fixed the issue in a later build, but I’m not clear why the Payara service cannot update to use this. Is it that all energy is on 7.x which uses the 6.x WELD?

[Pandrex247]

Payara 6 is still using Weld 5.0.x due to the removal of Weld Probe in 5.1 which we have built an integration for, making the update a breaking change for us until we rewrite the feature from scratch.

Payara 7 has simply ripped this integration out for now to allow us to update to an EE11 compatible version of Weld (6.0).

[marcreichman-pfi]

Thanks @Pandrex247 - so if I'm understanding correctly, this will never be fixed for 6.x? I was planning to update to 7.x as it is when it comes out, but I just wanted to understand if that's the official conclusion.

Thanks!

[Pandrex247]

I don't want to make any promises around Payara 6 Community as it will very likely be EoL at some point this year, but we are discussing what we intend to do about Weld in it and if we should simply bite the bullet and rip out the "CDI Dev Mode" integration without replacement to allow upgrading it to Weld 5.1 (since it's not really our feature, it's Weld's) vs. continuing to backport fixes to our patched 5.0 branch.

[marcreichman-pfi]

@Pandrex247 Thanks for your continued attention/time responding to this. In the end, I'm likely to go to community 7.x when it comes out, but until it does I am stuck on 6.2024.12 from this, and I know some other folks are in the same boat from the use of these instance patterns.

[lprimak]

Looks like #7482 should fix the issue. Just guessing however.

[marcreichman-pfi]

@lprimak @Pandrex247 @Viii3 If there's any way to generate a payara community 6.x build with this PR, i'd happily test it asap.

[lprimak]

@marcreichman-pfi Enjoy: https://nexus.hope.nyc.ny.us/repository/maven-releases/fish/payara/distributions/payara/6.2025.6-1/payara-6.2025.6-1.zip

[marcreichman-pfi]

@lprimak Thank you! I'll get on it.

[marcreichman-pfi]

It does appear this is helping! I'm going to run a better test over the weekend but I have no reason to suspect there's anything remaining that's a problem.

@pioneer2k can you give this a shot?

[marcreichman-pfi]

@bvschaik you may want to give this a shot too!

[pioneer2k]

@marcreichman-pfi as we are working with Payara Micro only I would need a patched version of that one.

[bvschaik]

@lprimak That version fixes the issue on the reproducer project.

[marcreichman-pfi]

@lprimak weekend testing looks good. Thanks!

[marcreichman-pfi]

@artur-mal @lprimak Can this be considered closed/fixed with https://github.com/payara/Payara/pull/7482 ?

[lprimak]

Yes