redun icon indicating copy to clipboard operation
redun copied to clipboard

How to isolate tasks in their own directory?

Open Asutu opened this issue 1 year ago • 0 comments

I'm trying to isolate a task in its own directory because it can generate a large number of files. I saw tempdir and thought that it could serve this purpose, but when I run the task with tempdir=True it returns an empty list, whereas by omitting this switch it does return all files.

The task:

@task(tempdir=True)
def splitter(data: File) -> List[File]:
    return script(
        f'''
        split -d -l 2 mydata.txt part.
        ''',
        inputs = [data.stage('mydata.txt')],
        outputs = [File(f).stage(str(f)) for f in glob_file('part.*')],
    )

Maybe tempdir=True has a different purpose? In any case, what is the recommended approach to isolate tasks in their own directory. (I'm running redun locally in my own laptop)

Thanks.

Asutu avatar Sep 18 '23 20:09 Asutu