Cannot pass arguments to test function in job submission

[SterlingButters]

I have the following code below that I am trying to use to pass argument 't' to the test function.

import socket
import time
import dispy


def test(t):
    time.sleep(t)
    print('\nSlept for {} seconds'.format(t))
    return 0


if __name__ == '__main__':
    # host = str(socket.gethostname())
    host = str(socket.gethostbyname(socket.gethostname()))
    cluster = dispy.JobCluster(test, nodes=[host], ip_addr=[host], depends=[])
    job = cluster.submit_node(host, 5)
    load = cluster.set_node_cpus(node=host, cpus=10)
    print('Job:', job())

    cluster.print_status()

Which returns:

Testing started at 10:12 PM ...
C:\ProgramData\Anaconda3\python.exe "C:\Program Files\JetBrains\PyCharm 2017.3\helpers\pycharm\_jb_pytest_runner.py" --path "C:/Users/Sterling Butters/PycharmProjects/DisPy/Test 2.py"
Launching py.test with arguments C:/Users/Sterling Butters/PycharmProjects/DisPy/Test 2.py in C:\Users\Sterling Butters\PycharmProjects\DisPy

============================= test session starts =============================
platform win32 -- Python 3.6.3, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
rootdir: C:\Users\Sterling Butters\PycharmProjects\DisPy, inifile:
collected 1 item
Test 2.py E
test setup failed
file C:\Users\Sterling Butters\PycharmProjects\DisPy\Test 2.py, line 6
  def test(t):
E       fixture 't' not found
>       available fixtures: cache, capfd, capsys, doctest_namespace, monkeypatch, pytestconfig, record_xml_property, recwarn, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

C:\Users\Sterling Butters\PycharmProjects\DisPy\Test 2.py:6


=================================== ERRORS ====================================
___________________________ ERROR at setup of test ____________________________
file C:\Users\Sterling Butters\PycharmProjects\DisPy\Test 2.py, line 6
  def test(t):
E       fixture 't' not found
>       available fixtures: cache, capfd, capsys, doctest_namespace, monkeypatch, pytestconfig, record_xml_property, recwarn, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

C:\Users\Sterling Butters\PycharmProjects\DisPy\Test 2.py:6
=========================== 1 error in 0.10 seconds ===========================
Process finished with exit code 0

yet if I change the test function to:

def test():
    t = 5
    time.sleep(t)
    print('\nSlept for {} seconds'.format(t))
    return 0

and remove the '5' in the cluster.submit_node(host, 5) then I get the following:

 Testing started at 10:19 PM ...
C:\ProgramData\Anaconda3\python.exe "C:\Program Files\JetBrains\PyCharm 2017.3\helpers\pycharm\_jb_pytest_runner.py" --target "Test 2.py::test"
Launching py.test with arguments Test 2.py::test in C:\Users\Sterling Butters\PycharmProjects\DisPy

============================= test session starts =============================
platform win32 -- Python 3.6.3, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
rootdir: C:\Users\Sterling Butters\PycharmProjects\DisPy, inifile:
collected 1 item
Test 2.py .
Slept for 5 seconds


========================== 1 passed in 5.08 seconds ===========================
Process finished with exit code 0

Am I missing something very trivial?

[pgiri]

submit_node should be called only after node is already initialized. In this case, it is likely that 'host' is not yet initialized. You can either wait a bit (e.g., with time.sleep before submit_node), or use cluster_status to know when a node is available for client to use.

[SterlingButters]

Thank you for such quick response! So regardless of the time value of time.sleep() I receive the same error in ~.1 seconds. Just to illustrate, I have:

...
    cluster = dispy.JobCluster(test, nodes=[host], ip_addr=[host], depends=[])
    time.sleep(10)
    job = cluster.submit_node(host, 5)
...

[pgiri]

In the first case the error message shows test setup failedbecause of 'fixture'. I am guessing this is due to the environment used (JetBrains / PyCharm etc.). I tested your program without and it works fine. If possible, you can try first with vanilla setup? It is likely PyCharm doesn't like arbitrary program fragments being evaled on the node?

[SterlingButters]

I get a different error that I was trying to avoid by using the host as the only node so I wouldn't have to worry about the ports as the different error I have been having also:

2018-01-20 23:04:43 pycos - version 4.6.5 with IOCP I/O notifier
2018-01-20 23:04:43 dispy - dispy client version: 4.8.4
2018-01-20 23:04:43 dispy - Storing fault recovery information in "_dispy_20180120230443"
2018-01-20 23:04:43 dispy - Error binding to port 51347: 10049 ...
2018-01-20 23:04:48 dispy - Invalid node
2018-01-20 23:04:48 dispy - Error binding to port 51347: 10049 ... 

Which I know the binding to port error comes from the requested address not being valid in its context--normally resulting from an attempt to bind to an address that is not valid for the local computer. So my question at that point would be whats wrong with my node? Can the host not be a node? Does the ip address have to follow an ipv4 format, or can it be 'DESKTOP-ICOTGB3' as mine is?

I've been messing around with this but to no avail and I haven't posted anything because I feel like I'm missing something incredibly obvious. I've already cleared the default ports in my firewall and that was all I could think of to do that might cause the issue.

[pgiri]

Host can be given as either IP address or hostname (in which case DNS is used to resolve it to IP address). In this case, it is likely node is not yet initialized. Instead of submit_node, you can try simpler case submit first (which would wait until node is initialized).

'Error binding to port' is likely due to another instance of dispy (client) still running. The error message can be confusing, because dispy allows for batch execution as well (so new instance waits for current instances to be finished).

[SterlingButters]

Trying the simpler case submit yields same binding error. I'm not sure what the best way to kill all instances of dispy but I included cluster.close() at the end of my code in hopes that would do the trick but no luck. How can I terminate instances or view existing ones?

[pgiri]

If you are using Windows, you can use 'TaskManager' to see running programs and kill appropriate process.

[SterlingButters]

As far as I can tell, when running the python code outside my IDE (vanilla) the only added app, background process, or windows process is the command prompt window (which is an 'app') which I am using to run the file. I assume closing this window ends associated processes that I can see, otherwise, I'm afraid to end processes that aren't obviously associated with dispy.

[SterlingButters]

If it helps, I get an error from a different cluster computing package: "[WinError 10054] An existing connection was forcibly closed by the remote host" ... not sure if the errors are correlated and it has something to do with my host?

[SterlingButters]

This may help most: I ran dispymode.py and received the following:

dispynode - node IP address fe80::4888:8c73:f9a2:10f2%7 seems to be loopback address; this will prevent communication with clients on other machines.

Is this the issue?

[pgiri]

Loopback address is definitely an issue. You can either give specific network address with -i option or install netifaces module (dispy will then avoid loopback and look for configured network address).