[BUG] Memgraph doesn't recognize python submodule update on `call mg.load_all()`

[antoniofilipovic]

Memgraph version Which version did you use?

Memgraph v2.0.0

Environment Some information about the environment you are using Memgraph on: operating system, how do you connect, with or without docker, which driver etc.

Operating system: Ubuntu 20.04, without docker, connecting with mgconsole 1.0

Describe the bug A clear and concise description of what the bug is. Python submodule doesn't update on call mg.load_all();

To Reproduce Steps to reproduce the behavior:

First, this is how I set up my folder structure

`python`
   test_module.py
   `mage`
      `test_submodule`
          __init__.py
         test_functions.py

This was my setup in /etc/memgraph/memgraph.conf --query-modules-directory=/home/antonio/work/memgraph/github-repos/field/github-repos/mage/python

Setup of test_functions.py

def test_function(a:int, b:int)->int:
    return a+b

Setup of test_module.py

import mgp

from mage.test_submodule.test_functions import test_function
@mgp.read_proc
def calculate(
    ctx: mgp.ProcCtx,
    a: mgp.Number,
    b: mgp.Number
) -> mgp.Record(result=mgp.Number):
    return mgp.Record(result=int(test_function(a,b)))

I started memgraph with following command:

antonio@antonio-ThinkPad-E480:~$ sudo runuser -l memgraph -c "/usr/lib/memgraph/memgraph"
You are running Memgraph v2.0.0

And then I started in the new terminal mgconsole and run the following commands:

antonio@antonio-ThinkPad-E480:~$ mgconsole --use-ssl=False
mgconsole 1.0
Connected to 'memgraph://127.0.0.1:7687'
Type :help for shell usage
Quit the shell by typing Ctrl-D(eof) or :quit
memgraph> call test_module.calculate(2,3) YIELD result RETURN result;
+--------+
| result |
+--------+
| 5      |
+--------+
1 row in set (0.052 sec)
memgraph> 

The result till now was as expected. But then I change what test_function to return a*b instead of a+b

New setup of test_functions.py

def test_function(a:int, b:int)->int:
    return a*b

And run these commands in mgconsole:

memgraph> call mg.load_all();
Empty set (0.075 sec)
memgraph> call test_module.calculate(2,3) YIELD result RETURN result;
+--------+
| result |
+--------+
| 5      |
+--------+
1 row in set (0.047 sec)

Expected behavior A clear and concise description of what you expected to happen. I expected the result to be 6.

Further note: I tried to add one more function in my test_functions.py

def test_function(a:int, b:int)->int:
    return a*b
def test_function2(a:int, b:int)->int:
    return a*b

And then changed test_module.py

import mgp

from mage.test_submodule.test_functions import test_function, test_function2


@mgp.read_proc
def calculate(
    ctx: mgp.ProcCtx,
    a: mgp.Number,
    b: mgp.Number
) -> mgp.Record(result=mgp.Number):
    return mgp.Record(result=int(test_function2(a,b)))

But got following result:

memgraph> call mg.load_all();
Empty set (0.093 sec)
memgraph> call test_module.calculate(2,3) YIELD result RETURN result;
Client received exception: There is no procedure named 'test_module.calculate'.
memgraph> 

And logs:

[2021-10-15 13:00:59.495] [memgraph_log] [info] Loading module "/home/antonio/work/memgraph/github-repos/field/github-repos/mage/python/test_module.py"...
[2021-10-15 13:00:59.496] [memgraph_log] [error] Unable to load module "/home/antonio/work/memgraph/github-repos/field/github-repos/mage/python/test_module.py"; Traceback (most recent call last):
  File "/home/antonio/work/memgraph/github-repos/field/github-repos/mage/python/test_module.py", line 3, in <module>
    from mage.test_submodule.test_functions import test_function, test_function2
ImportError: cannot import name 'test_function2' from 'mage.test_submodule.test_functions' (/home/antonio/work/memgraph/github-repos/field/github-repos/mage/python/mage/test_submodule/test_functions.py)

Logs

These are logs for problem for changing a+b to a*b If applicable, add logs of Memgraph, CLI output or screenshots to help explain your problem.

[2021-10-15 12:49:11.941] [memgraph_log] [debug] [Run] 'call test_module.calculate(2,3) YIELD result RETURN result'
[2021-10-15 12:52:25.618] [memgraph_log] [debug] [Run] 'call mg.load_all()'
....
[2021-10-15 12:52:25.620] [memgraph_log] [info] Closing module "/home/antonio/work/memgraph/github-repos/field/github-repos/mage/python/test_module.py"...
[2021-10-15 12:52:25.621] [memgraph_log] [info] Closed module "/home/antonio/work/memgraph/github-repos/field/github-repos/mage/python/test_module.py"
....

...
[2021-10-15 12:52:25.686] [memgraph_log] [info] Loading module "/home/antonio/work/memgraph/github-repos/field/github-repos/mage/python/test_module.py"...
[2021-10-15 12:52:25.687] [memgraph_log] [info] Loaded module "/home/antonio/work/memgraph/github-repos/field/github-repos/mage/python/test_module.py"

Additional context Add any other context about the problem here.

Issue also mentioned on Orbicon: https://github.com/memgraph/orbicon/issues/26 I found out now that this issue is mention also here: https://github.com/memgraph/memgraph/issues/223#issue-985222353

[jbajic]

I reproduced the same behaviour, memgraph doesn't reload the imported modules from test_function module, and does not register the changes with mg.load_all() call. The quick fix would fix from your side would be to call reload manually in the code, something like this:

import importlib
import mgp
import mage.test_submodule.test_functions as test_functions

@mgp.read_proc
def calculate(
    ctx: mgp.ProcCtx,
    a: mgp.Number,
    b: mgp.Number
) -> mgp.Record(result=mgp.Number):
    importlib.reload(test_functions)
    return mgp.Record(result=int(test_functions.test_function(a,b)))

or you can write those utility functions where you wrote the mg procedure, then the changes will be reloaded.

[antonio2368]

Maybe we can get a list of all imported modules or just the imports for the module in question and reload them ourselves. Not sure if it's possible we would need to check it out so above solution seems the best.

[antoniofilipovic]

I think importlib.reload() is not the best solution, if it could be solved inside Memgraph it would be awesome.

[antonio2368]

The problem is that Python uses sys.modules to cache the loaded modules. A basic solution could delete all the cached modules on mg.load_all and reload everything. But this could have some weird side effects if it's used with mg.load (what if you reload a module used by some other module). Is that still okay?

[gitbuda]

@antoniofilipovic is this still an issue?

[antoniofilipovic]

Yes, it is. If I am correct we haven't done anything on this side to fix this bug.

[gitbuda]

@as51340 this is related to you current work 😄