Excessive memory consumption with Azure storage backend

[adejanovski]

Memory consumption of Medusa with the Azure storage backend can grow excessively to 10s of GB. The behavior was noted to be more important when the azcli is used over libcloud.

It looks like the --content-md5 argument was used for each azcli invocation, with a computation of the hash made by Medusa through a static method. It's safe to infer that computing the hash for each uploaded file can cause such memory consumption, which could leak and not be garbage collected.

┆Issue is synchronized with this Jira Task by Unito ┆friendlyId: K8SSAND-1600 ┆priority: Medium

[renoypaulose]

@adejanovski Did you get a chance to work on this item? This is the memory usage by medusa in our k8ssandra cluster for 24hrs duration. we are taking incremental backup every 4hrs and we are killing medusa once a day(around 02:40) to clear the memory.

[renoypaulose]

@adejanovski Any update on this issue?

[rzvoncek]

Hello,

We overhauled the storage backends and it's likely this is no longer an issue.

If still relevant, could we re-try with medusa v 0.16.1 to see if this is still valid?

[rzvoncek]

Closing this because it's likely fixed. Please let us know if that's not the case.