Fail to use multiprocessing from map function

[kpavel]

Currently the following code fails on pickle when trying to invoke the pool.map from inside of the map function

#!/usr/bin/python

import lithops
import multiprocessing

def mult(val):
    print(val)
    return val * val

def testPool(vals):
    pool = multiprocessing.Pool(8)
    return pool.map(mult, vals)

fexec = lithops.FunctionExecutor()
fexec.call_async(testPool, [1, 2, 3])
print(fexec.get_result())

the stacktrace is:

Traceback (most recent call last):
  File "/action/lithops/worker/jobrunner.py", line 278, in run
  File "poolTest.py", line 19, in testPool
    return pool.map(mult, vals)
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 364, in map
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 771, in get
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 537, in _handle_tasks
  File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 206, in send
  File "/usr/local/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
_pickle.PicklingError: Can't pickle <function mult at 0x7fcf0dd8a790>: attribute lookup mult on __main__ failed

@JosepSampe can you please take a look?

[JosepSampe]

I started investigating this and seems that the problem is located in the runtime, and not in lithops itself.

It needs further investigation, but for now what I did is to create a simple function in the cloud using the ibmcloud cli, and test it with different runtimes. The code is the next (as you can se it is not lithops related):

import sys
import multiprocessing

def mult(val):
    return val * val

def testPool(vals):
    pool = multiprocessing.Pool(8)
    result = pool.map(mult, vals)
    print(result)

def main(dict):
    testPool([1, 2, 3])
    return { 'message': 'Hello world' }

Then I created 3 different actions with different runtimes (and python versions):

ic fn action update test_mp_36 test_mp.py --docker ibmfunctions/action-python-v3.6
ic fn action update test_mp_37 test_mp.py --docker ibmfunctions/action-python-v3.7
ic fn action update test_mp_38 test_mp.py --docker jsampe/action-python-v3.8

Only the runtime with Python3.7 executes the action correctly, the rest of actions fail with the error message you posted above.

[JosepSampe]

Closing it for now since it is an issue inherited from python and not from Lithops. If a user wants to use the multiprocessing library inside a lithops function, the function must be defined in a separate file.