module functions and variables not found

[dnsmkl]

Parallel stream raises NameError for anything defined at module level.

System/Version:

• pyfunctional==1.4.3
• windows 10 (same problem does not occur on linux)

Minimal reproducing code:

    import functional
    
    def myprint(arg):
        print(arg)

    if __name__ == "__main__":
        pseq = functional.streams.ParallelStream(processes=1)
        pseq(1,2,3).map(myprint).all() # works
        pseq(4,5,6).map(lambda x: myprint(x)).all() # fails - NameError: name 'myprint' is not defined

Expected output:

  1
  2
  3
  4
  5
  6

Actual output:

  1
  2
  3
  multiprocessing.pool.RemoteTraceback: 
  """
  Traceback (most recent call last):
    File "c:\Program Files\Python311\Lib\multiprocessing\pool.py", line 125, in worker
      result = (True, func(*args, **kwds))
                      ^^^^^^^^^^^^^^^^^^^
    File "C:\reproduce\venv\Lib\site-packages\functional\util.py", line 121, in unpack
      return list(result)
             ^^^^^^^^^^^^
    File "C:\reproduce\app\main.py", line 9, in <lambda>
      pseq(4,5,6).map(lambda x: myprint(x)).all() # fails - NameError: name 'myprint' is not defined
                                ^^^^^^^
  NameError: name 'myprint' is not defined
  """
  
  The above exception was the direct cause of the following exception:
  
  Traceback (most recent call last):
    File "C:\reproduce\app\main.py", line 9, in <module>
      pseq(4,5,6).map(lambda x: myprint(x)).all() # fails - NameError: name 'myprint' is not defined
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\reproduce\venv\Lib\site-packages\functional\pipeline.py", line 675, in all
      return all(self)
             ^^^^^^^^^
    File "C:\reproduce\venv\Lib\site-packages\functional\util.py", line 167, in lazy_parallelize
      for pool_result in pool.imap(unpack, packed_partitions):
    File "c:\Program Files\Python311\Lib\multiprocessing\pool.py", line 873, in next
      raise value
  NameError: name 'myprint' is not defined
  
  Process finished with exit code 1

It also fails when referencing module level import, or variable, or function that uses other function:

    import functional
    import os
    def printarg(arg): print(arg)
    def chain_f(arg): printarg(arg)
    MYCONST = 1
    def printconst(arg): print(MYCONST)
    def printosname(arg): print(os.name)
    if __name__ == "__main__":
        pseq = functional.streams.ParallelStream()
        pseq(1).map(printarg).all() # works
        # and the rest fail:
        pseq(2).map(printconst).all() # NameError: name 'MYCONST' is not defined
        pseq(3).map(lambda x: print(MYCONST)).all() # NameError: name 'MYCONST' is not defined
        pseq(4).map(chain_f).all() # NameError: name 'printarg' is not defined
        pseq(5).map(lambda x: printarg(x)).all() # NameError: name 'printarg' is not defined
        pseq(6).map(printosname).all() # NameError: name 'os' is not defined
        pseq(7).map(lambda x: print(os.name)).all() # NameError: name 'os' is not defined

[artemisart]

This could be an issue/particularities of python multiprocessing on windows, not pyfunctional. Can you try wrapping your code in a def main function and calling from the if __main__ ?

[Yangeok]

Is pseq working in ipykernel file?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[EntilZha]

I'd agree with @artemisart, this seems like a namespace issue with parallel computation. Could you check if using the multiprocessing modules results in something similar? In general, a good idea to have a main method anyway since otherwise things in the __main__ block are module scope.

@Yangeok not sure what you mean.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.