bluesky
Fix memory leak
We've known for a couple weeks about a clear memory leak in Tiled. I don't have a simple reproducible example to recreate it, and my initial attempts to write one have failed, so it may not appear in every situation. But we see in the production NSLS2 deployment and @tacaswell has reproduced it locally on a much smaller deployment.
Using a beamline database that uses databroker to construct datasets from documents, I was able to find a caching issue that led to runaway memory usage. the fix is shown in the following pull request: https://github.com/bluesky/databroker/pull/741
The linked fix seems to address the issue we observed in production at NSLS2. I will leave this issue open while we look into @tacaswell’s report, which did not involve Databroker and seems to be a separate issue.
@tacaswell shared an example which he believed would reproduce the issue he saw. (The materials are too large to upload here.) I have tried reproducing on latest (v0.1.0a91) and on my best guess at what he was running at the time of the original verbal report, based on the approximate date (v0.1.0a74). I could not reproduce.
I am going to close this because there is nothing actionable, but I am highly interested in reproducible reports of memory usage growing over time when the server is at rest or under load.
It's worth noting that @tacaswell reproduced this in his build-bleeding-edge-CPython-and-all-libraries environment, so something in there could have been a factor here. Reason enough to wait for someone to reproduce.