Possible memory leak with Twitcher/Magpie

[tlvu]

Describe the bug

Twitcher is constantly taking lots of Memory.

Note below Twitcher is taking 23G of memory, while the front proxy only 30M, while having similar "NET I/O".

$ docker stats proxy twitcher thredds
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
fad5cec748b9        proxy               13.83%              30.29MiB / 125.8GiB   0.02%               4.29TB / 4.34TB     0B / 0B             10
4612ca2a1be0        twitcher            17.63%              23.73GiB / 125.8GiB   18.86%              4.82TB / 4.83TB     0B / 0B             51
81aaf9dcbdc8        thredds             118.74%             7.014GiB / 125.8GiB   5.58%               26.6GB / 4.25TB     0B / 0B             104

We have had a lot of Thredds activity lately so it's normal for the Cpu and Memory consumption to increase for Thredds since it has a caching feature.

But Twitcher saw a proportionally increase which is puzzling to me and stay like this during idle time.

Related bug on Magpie https://github.com/Ouranosinc/Magpie/issues/505

To Reproduce

PAVICS deployed at this commit from pour production fork of birdhouse-deploy https://github.com/Ouranosinc/birdhouse-deploy/commit/76dd3c86e57d6a96ca84d2124f3b019fe278645a which triple Thredds memory.

Lots of Thedds OpenDAP access on large .ncml file.

Expected behavior

Twitcher Cpu and Memory would increase during Thredds transfer/activity but should go down during idle time, just like the proxy container.

Context

• Instance: https://pavics.ouranos.ca/twitcher/
• Version: [{"name": "Twitcher", "version": "0.6.2", "type": "application"}, {"name": "magpie.adapter.MagpieAdapter", "version": "3.19.1", "type": "adapter"}]