Possible memory leak?

[eager-seeker]

• SPADE version: 3.1.4
• Python version: 3.7.4
• Operating System: Mac OS Mojave
• IM Server: prosody version 0.9.14

Description

Agents repeatedly sending and receiving messages via behaviours eventually seem to cause memory leakage.

What I Did

When trying to scale numbers of agents, or messages per agent in multi-agent experimentation, performance always degrades quite quickly and memory-use steadily increases. Garbage collection doesn't seem to recover it. When trying to debug using a minimal agent that was just being prompted to add repeated OneShotBehaviour to send inform messages, it could be seen that, for example, JID objects were being created and didn't seem to get garbage collected. Is the way the synchronous code is calling add_behaviour allowed?

# try this also with mprof: 
# mprof run --include-children python myscript.py
# mprof plot --output mem-plot.png
import objgraph
import gc
import time

from spade import quit_spade
from spade.agent import Agent
from spade.behaviour import OneShotBehaviour
from spade.message import Message

import test_creds # Simple module with just a variable declared: pwd

def jid_filter(x):
    return (type(x).__name__ == "JID")

class SenderAgent(Agent):
    class InformBehav(OneShotBehaviour):
        async def run(self):
            msg = Message(to=self.agent.recv_jid)  # Instantiate the message
            msg.set_metadata("performative", "inform")  # Set the "inform" FIPA performative
            msg.body = "Hello World {}".format(self.agent.recv_jid)  # Set the message content
            await self.send(msg)


    def send_inform(self):
        b = self.InformBehav()
        self.add_behaviour(b)

    def __init__(self, recv_jid, *args, **kwargs):
        self.recv_jid = recv_jid
        super().__init__(*args, **kwargs)

def dump_garbage():
    print("gc.collect()")
    gc.collect()

    print("*** GARBAGE DUMP:")
    for x in gc.garbage:
        s = str(x)
        if len(s) > 80: s = s[:77]+'...'
        print(type(x),"\n  ", s)

if __name__ == "__main__":
    # gc.set_debug(gc.DEBUG_LEAK)  # uncomment to debug
    print("*** About to start main...")
    objgraph.show_growth(limit=10, filter=jid_filter)   # Stop and show change

    sender_jid = f"agent0@localhost/dummy"
    recv_jid = f"agent1@localhost/dummy"
    sender_passwd = test_creds.pwd

    senderagent = SenderAgent(recv_jid, sender_jid, sender_passwd)
    future = senderagent.start(auto_register=True)
    future.result()
    print(f"Sender {sender_jid} started")
    objgraph.show_growth(limit=10, filter=jid_filter)

    for step in range(24000):
        time.sleep(0.1)
        senderagent.send_inform()
        if step % 600 == 0:
            print(f"*** Completed step {step}...")
            objgraph.show_growth(limit=5)
            gc.collect()

    for _ in range(10):
        try:
            time.sleep(1)
            print(".", end="")
        except KeyboardInterrupt:
            break

    print("#####     Agents finished")
    objgraph.show_most_common_types(limit=20, filter=jid_filter)

    # dump_garbage()  # uncomment to debug

    quit_spade()
    print("#####     Quit spade")
    objgraph.show_most_common_types(limit=5)
    # dump_garbage()  # uncomment to debug
    print("#####     Slán")

[javipalanca]

I guess this memory leak is due to the large list of behaviors added to the agent, since the behaviors are not deleted from the list when they are done, which will be fixed in the next version. Thanks for identifying the problem!

[eager-seeker]

I did spot that a while back, and in other code I had a routine to try and clear out dead behaviours. But even with that in I seem to be racking up a huge number of object instances:

    def clear_dead_behaviours(self):
        for behave in self.behaviours:
            if behave.is_killed():
                self.behaviours.remove(behave)

[dmvieira]

I saw same problem here