AttributeError: Can't pickle local object 'keras_modules_injection.<locals>.wrapper'

[fcarsten]

Environment:

Windows 10 Home, 64Bit Python 3.6 (also happens with 3.7) TensorFlow 2.1 (also happens with 2.0, happens with both CPU or GPU processing).

I'm trying to run the code directly, i.e. not using Docker, with this code (CPU version):

from evaluater.predict import main

if __name__ == '__main__':
    main("MobileNet", "../models/MobileNet/weights_mobilenet_aesthetic_0.07.hdf5", "../readme_figures/images_aesthetic", None)

This crashes with the following error message:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\carsten\AppData\Local\Programs\Python\Python36\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\carsten\AppData\Local\Programs\Python\Python36\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\carsten\PythonVenvs\image-quality-assessment-3-6\lib\site-packages\tensorflow_core\python\keras\utils\data_utils.py", line 844, in _run
    with closing(self.executor_fn(_SHARED_SEQUENCES)) as executor:
  File "C:\Users\carsten\PythonVenvs\image-quality-assessment-3-6\lib\site-packages\tensorflow_core\python\keras\utils\data_utils.py", line 823, in pool_fn
    initargs=(seqs, None, get_worker_id_queue()))
  File "C:\Users\carsten\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 119, in Pool
    context=self.get_context())
  File "C:\Users\carsten\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 174, in __init__
    self._repopulate_pool()
  File "C:\Users\carsten\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 239, in _repopulate_pool
    w.start()
  File "C:\Users\carsten\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\carsten\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\carsten\AppData\Local\Programs\Python\Python36\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\carsten\AppData\Local\Programs\Python\Python36\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'keras_modules_injection.<locals>.wrapper'

2020-03-07 22:21:57.571879: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\carsten\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\carsten\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

If I change predict.py line 29ff to disable multiprocessing:

def predict(model, data_generator):
    return model.predict(data_generator, workers=8, use_multiprocessing=False, verbose=1)

The code runs fine.