Create temporary directory per user?

[jonasc]

I just stumbled upon the file structure and thought about what will happen if two users run dispy on the same machine. If I understand correctly you create /tmp/dispy/node and set it's access right to 700 so no other user is allowed to see or even write into this directory.

If this is correct, a second user trying to run dispynode will get this message (just tested):

Traceback (most recent call last):
  File "/home/test/.virtualenvs/dispy/bin/dispynode.py", line 1719, in <module>
    _dispy_node = _DispyNode(**_dispy_config)
  File "/home/test/.virtualenvs/dispy/bin/dispynode.py", line 302, in __init__
    with open(os.path.join(self.dest_path_prefix, 'config'), 'wb') as fd:
PermissionError: [Errno 13] Permission denied: '/tmp/dispy/node/config'

Wouldn't it be better to create e.g. /tmp/dispy_<username>/node or /tmp/dispy/<username>/node?

[pgiri]

A node should run only one instance of dispynode, as it assumes it can schedule all processors available. If another instance runs simultaneously, they can over-schedule processors. Ideally file locking should be used to prevent multiple instances, but not sure how easy / portable it is for multiple architectures.

I am curious to know use case where different users want to run dispynode simultaneously? It may be possible to setup paths as you suggest (or perhaps /tmp/dispy/node// where is process id of dispynode).

[jonasc]

Your explanation sounds reasonable. However I think this might come on in Situations like mine: I'm using various university computers as dispynodes to speed up some computations. If someone else has the same idea they could use the nodes I set up but they might not for (at least) two reasons:

1. I protect the nodes with authentification or ssl.
2. They do not want to run computations on nodes not under their control.

In the end they might set up nodes themselves with differing port numbers.

[pgiri]

Multiple instances of dispynode can be run by specifying different ports and different --dest_path_prefix options. While it is more cumbersome to use --dest_path_prefix, it may prevent accidental "abuse". May be the right thing is to provision portion of processors in each instance (e..g., user 1 uses 1/4 of available processors, user 2 remaining etc.)? I think this shouldn't interfere with rest of dispy, other than probably some spurious warning messages about user 1's client not being able to use user 2's server etc.

[jonasc]

I guess, since it is no real issue for me now and was rather just thinking theoretically about it, you could just leave it as is. If someone stumbles across this issue they might also have some sensible ideas on how to solve.

[yschroeder]

I ran into the described problem. I told a colleague about dispy and he tried in on one of our machines. He stopped is test but the folder /tmp/dispy/ was created for him. This rendered me unable to use that machine for my instance of dispynode.py.

I believe dispy should create folders in /tmp/ with an appended random string or something (like most other software does to handle this kind of problem). Maybe the temporary directory should be created with this function: https://docs.python.org/3.6/library/tempfile.html#tempfile.mkdtemp

[yschroeder]

Same problem arises when running dispyscheduler.