yagna icon indicating copy to clipboard operation
yagna copied to clipboard

Provider agent memory leak

Open mfranciszkiewicz opened this issue 2 years ago • 3 comments

Investigate and fix a memory leak in the provider agent.

image

host: office_8_67 net: devnet-beta

Max 1 day for investigating

mfranciszkiewicz avatar May 17 '22 14:05 mfranciszkiewicz

This is still happening, although so far only observed on devnet-beta providers, running yagna 0.10.0.

I suspect this is related to #1925.

I saw this correlation here: https://github.com/golemfactory/golem-internal/issues/497 graph

and here: Screenshot_20220905_095547

The rate of memory leaking is quite different, but in both cases I can see it's burning 100% CPU (one vcore), while looping on an event.

The issue #1925 is already resolved, but remaining question is: can someone intentionally generate a lot of events to perform a DoS attack?

etam avatar Sep 05 '22 08:09 etam

This issue was not related to someone generating a lot of events, but to problem with sqlite timestamps comparison, which caused this effect. So even if it is possible to cause DoS in some way, it is not by exploiting problems discussed here.

Can you confirm, that this issue isn't happening anymore?

nieznanysprawiciel avatar Sep 05 '22 10:09 nieznanysprawiciel

I haven't noticed it in recently in hybrid network (it's not a 100% firm "no", but it's also not a "yes, it's still happening"). Some nodes are filling their disk space, but due to smoltcp::socket::tcp and ya_net::hybrid::service debug messages.

Edit: also provider_datadir/exe-unit/work takes a lot of space.

etam avatar Sep 05 '22 11:09 etam