taskserver
taskserver copied to clipboard
GothenburgBitFactory
[TD-77] taskd sudden memory bloat
Jack on 2014-10-16T17:01:09Z says:
Some time around the beginning of October, taskd's memory usage on freecinc would occasionally spike to consume all available system memory, and no clients would be able to sync.
The logs always seem to end with a successful sync, but given TD-72 this is not surprising. Since nothing appears to be logged until the end of a sync, I haven't identified a way to determine wither a particular account is the cause of the memory spike.
When this memory spike occurs, the memory usage grows to 100MB or more within the course of one minute, and will continue to consume all available memory.
The situation has been abated by monitoring the taskd process and restarting it if memory grows beyond a particular threshold.
The last known occurrences of this bug were once on Oct 7th 2014 and once on Oct 8 2014.
Migrated metadata:
Created: 2014-10-16T17:01:09Z
Modified: 2015-04-27T17:12:35Z
Renato Alves on 2014-10-16T17:48:24Z says:
Hi Jack, does memory remain high once clients disconnect?
Renato Alves on 2014-10-16T22:44:05Z says:
Attached is a test that reproduces a slow memory leak.
I was not able to reproduce the "sudden" effect but it does rise quite fast during a sync of big chunks of data are being synced.
Though if the chunks are above 1M (default) the server will simply respond with "504 Request too big".
Jack on 2014-10-16T23:50:27Z says:
Renato,
I don't have a way of proving that any clients are indeed connected. To my knowledge it has never recovered from this condition, except by restarting taskd.
Paul Beckingham on 2015-02-28T18:21:57Z says:
Renato, how would you propose we get td-77.t into play? Clone the test framework into td? Put td-77.t into tw?
Renato Alves on 2015-02-28T21:54:10Z says:
Paul,
I'm not sure this test would be a good candidate for merging. It serves to reproduce the problem but does so in a somewhat crude way. It also takes forever to run...
What I've been doing to run this test is to symlink the taskd binary to tw/src/ and put the test in tw/test/.
As for having tests and the framework in the taskd repo, I'd say cloning is the simplest. Downside being that discipline will be needed to ensure both repositories have their frameworks in sync. We excluded git submodules in a previous discussion. We also mentioned using a virtualenv but this would require additional setup to the current testing infrastructure.
Paul Beckingham on 2015-02-28T22:08:05Z says:
Okay, thanks, then we keep td-77.t to help us fix this only. For now.
Paul Beckingham on 2015-04-25T16:32:49Z says:
Reducing this from 'blocker' to 'critical'. I have no idea how to replicate or address this. Yet.
Renato, the td-77.t is not working in taskd.git/test dir, I'm going to need your help with this.
Paul Beckingham on 2015-04-26T16:23:47Z says:
Jack, can you provide your GnuTLS version? We're seeing improvements in version 3.3.9 and later.
Jack on 2015-04-27T16:58:02Z says:
Does this provide the information you are looking for?
{code} $ gnutls-cli --version gnutls-cli (GnuTLS) 2.12.23 Packaged by Debian (2.12.23-12ubuntu2.1) Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
Written by Nikos Mavrogiannopoulos. {code}
Paul Beckingham on 2015-04-27T17:12:35Z says:
Perfect, thanks. It's consistent with what we see elsewhere.
GnuTLS leaks, a lot, but those leaks are being fixed, and you can see the commits made in gnutls.git to address these.
Simply put, upgrade GnuTLS for leak mitigation, and security fixes. Not a great answer I'm afraid, but it's the truth.
