NoneType Error

[chunweizhu]

dflow-helloworld.ipynb

from https://github.com/zjgemi/dflow_helloworld/blob/master/dflow-helloworld.ipynb

#Define a Python OP
from pathlib import Path
from dflow.python import OP, Artifact
from dflow import SlurmRemoteExecutor

slurm_remote_executor = SlurmRemoteExecutor(
    host= "xxx.xx.xxx.xx", 
    port= xxx, 
    username="xxxxxx",
    private_key_file="D:\\fsdownload\\rid_rsa", 
    header="#!/bin/bash\n#BATCH -J job\n#SBATCH -p test_q -N 1 -n 12\nsource activate py37" 
    )

@OP.function
def duplicate(num: int, foo: Artifact(Path)) -> {"num": int, "bar": Artifact(Path)}:
    #print("foo Here==",foo)
    with open(foo, "r") as f:
        content = f.read()
    with open("bar.txt", "w") as f:
        f.write(content * 2)
    return {
        "num": num * 2,
        "bar": Path("bar.txt"),  #输出文件
    }

from dflow import upload_artifact
with open("foo.txt", "w") as f:
    f.write("Hellooooooooooooooooooooooo")
art = upload_artifact("foo.txt")
#print("art==",art,type(art))

from dflow.python import PythonOPTemplate
from dflow import Step

step1 = Step(
    name="step1",
    template=PythonOPTemplate(duplicate, image="python:3.7"),
    parameters={"num": 1},
    artifacts={"foo": art}, 
    executor=slurm_remote_executor,
)

step2 = Step(
    name="step2",
    template=PythonOPTemplate(duplicate, image="python:3.7"),
    parameters={"num": step1.outputs.parameters["num"]}, #使用第一个Step的输出参数和文件
    artifacts={"foo": step1.outputs.artifacts["bar"]},
    executor=slurm_remote_executor,
)

#Add step into a workflow
from dflow import Workflow

wf = Workflow(name="python-test")
wf.add(step1)
wf.add(step2)

#Submit the workflow
wf.submit()

Error

Warning: Permanently added 'xxx.xx.xxx.xx' (ECDSA) to the list of known hosts.
Submitted batch job 181941
Traceback (most recent call last):
  File "script", line 89, in <module>
    output = op_obj.execute(input)
  File "~/.conda/envs/py37/lib/python3.7/site-packages/dflow/python/op.py", line 118, in wrapper_exec
    op_out = func(self, op_in)
  File "script", line 53, in execute
    with open(op_in["foo"], "r") as f:
TypeError: expected str, bytes or os.PathLike object, not NoneType

[zjgemi]

SlurmRemoteExecutor has not been maintained for a while. Can you try to use DispatcherExecutor instead of SlurmRemoteExecutor? E.g.

from dflow.plugins.dispatcher import DispatcherExecutor

dispatcher_executor = DispatcherExecutor(
    host= "xxx.xx.xxx.xx", 
    port= xxx, 
    username="xxxxxx",
    private_key_file="D:\\fsdownload\\rid_rsa", 
    resources_dict={"custom_flags": ["#!/bin/bash", "#BATCH -J job", "#SBATCH -p test_q -N 1 -n 12", "source activate py37"]},
    )

[chunweizhu]

I tried it, but a new problem appeared ^-^：

slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: time="2024-01-04T01:41:48 UTC" level=info msg="capturing logs" argo=true slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: 2024-01-04 01:41:51,753 - INFO : info:check_all_finished: False slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: 2024-01-04 01:41:51,753 - INFO : remote path: /home/dflow/workflows/5fa58bc8635b4ee628d0c00bcf1fbd4d63243d6c slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: Traceback (most recent call last): slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: File "/argo/staging/script", line 115, in slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: submission.run_submission(clean=True) slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: File "/opt/venv/lib/python3.12/site-packages/dpdispatcher/submission.py", line 226, in run_submission slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: self.upload_jobs() slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: File "/opt/venv/lib/python3.12/site-packages/dpdispatcher/submission.py", line 495, in upload_jobs slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: self.machine.context.upload(self) slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: File "/opt/venv/lib/python3.12/site-packages/dpdispatcher/contexts/ssh_context.py", line 584, in upload slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: self.ssh_session.sftp.chdir(self.temp_remote_root) slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: File "/opt/venv/lib/python3.12/site-packages/paramiko/sftp_client.py", line 659, in chdir slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: if not stat.S_ISDIR(self.stat(path).st_mode): slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: ^^^^^^^^^^^^^^^ slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: File "/opt/venv/lib/python3.12/site-packages/paramiko/sftp_client.py", line 493, in stat slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: t, msg = self._request(CMD_STAT, path) slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: File "/opt/venv/lib/python3.12/site-packages/paramiko/sftp_client.py", line 857, in _request slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: return self._read_response(num) slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: ^^^^^^^^^^^^^^^^^^^^^^^^ slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: File "/opt/venv/lib/python3.12/site-packages/paramiko/sftp_client.py", line 909, in _read_response slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: self._convert_status(msg) slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: File "/opt/venv/lib/python3.12/site-packages/paramiko/sftp_client.py", line 938, in _convert_status slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: raise IOError(errno.ENOENT, text) slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: FileNotFoundError: [Errno 2] No such file slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: time="2024-01-04T01:41:52 UTC" level=info msg="sub-process exited" argo=true error="" slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: time="2024-01-04T01:41:52 UTC" level=error msg="cannot save parameter /tmp/outputs/parameters/msg" argo=true error="open /tmp/outputs/parameters/msg: no such file or directory" slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: time="2024-01-04T01:41:52 UTC" level=warning msg="cannot save artifact /tmp/outputs/artifacts/bar" argo=true error="stat /tmp/outputs/artifacts/bar: no such file or directory" slurm-f8v9c-duplicate-qm6pv-jaztc-3072193478: Error: exit status 1

[zjgemi]

Make sure the remote root (default: /home/{username}/dflow/workflows) exists on the remote cluster. The remote root can be specified as an argument of DispatcherExecutor.