Possible memory leak on very long running `dagster-webserver`/Daemon containers

[cody-scott]

Dagster version

1.7.1

What's the issue?

Web UI and Daemon when running for a long period of time (months) without restarting, progressively uses additional memory until restarted.

Example here is we had an older version of dagster running (1.6.0) since February. When we updated to the newest version this week it released a large amount of memory and recovered back up to the original starting point.

Running OSS on Ubuntu in docker on a single machine.

What did you expect to happen?

No response

How to reproduce?

No response

Deployment type

None

Deployment details

Ubuntu 22.04 docker compose for webserver + daemon images

Additional information

Unclear if this is a docker issue holding on to memory, or if this is a dagster daemon/webserver holding on. Will let it run again to see if it appears.

Additionally, is the best practice to frequently restart the webserver/daemons to avoid this.

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[garethbrickman]

This could be a duplicate of https://github.com/dagster-io/dagster/issues/18997, and the solution in there is to downgrade and pin the grpcio python package to 1.57.0.

[cody-scott]

Perfect! I'll try it out next week.

[shivonchain]

@garethbrickman, the link you referenced points to an internal repo, is there a publicly accessible PR to track progress on this? Thanks!

[garethbrickman]

@shivonchain Ah my apologies I linked that erroneously and have removed it. It was not a fix but an internal test.