Distributed single task execution

[amarzullo24]

Dear all, I have a question regarding the execution of single tasks in different machines running Pydra. More specifically, I was wondering whether tasks requiring high computational power can be distributed to more powerful (HPC) machines.

Reading the API documentation and following the interactive tutorial, I understood that an experimental distributed execution of single tasks could be achieved by using ConcurrentFutures, SLURM or Dask. However, I was not able to find any specific example about how to use it.

By setting the parameter of the Submitter plugin="dask", for example, a DaskWorker can be instantiated. However, is not clear whether this can be useful to achieve my goal.

Thanks in advance for your help!

[djarecka]

Hi @emmeduz, thank you for your question. DaskWorker after recent changes is not well tested, but you should be able to use ConcurentFutures and Slurm. The task are automatically distributed for task wit splitter (see examples here) and for Workflows (see example here). If some examples do not specify any plugin then ConcurentFutures (or cf) is used automatically.

Please let me know if this answer your question

[amarzullo24]

Dear @djarecka, thank you for your prompt reply! I have practiced with those examples, however it is not yet clear how the Slurm or ConcurrentFutures plugin can be exploited to distribute the task on different machines (e.g. I cannot find any example about how to specify the destination address).

I was studied a bit Dask, even if I know it is not well tested yet, and I found out a workaround as follows:

import nest_asyncio
import pydra
from dask.distributed import Client, LocalCluster

nest_asyncio.apply()

# define a pydra task
@pydra.mark.task

def f(x):
   return x + 1

task1 = f(x=0)

# create a Submitter using Dask as backend
with pydra.Submitter(plugin="dask", address='127.0.0.1:1234') as sub:
   sub(task1)

# gather the task execution results
task1.result()

Do you think is it possible doing something similar using Slurm or ConcurrentFutures?

Thanks for all your help!