`task`: optionally download data on `refresh`

[DavidGOrtega]

Im missing the capability to retrieve the intermediate data at any point. Maybe we could add a flag (resource property) like pullOnRefresh

[0x2b3bfa0]

🙈 I wouldn't want to pull data every time I want to check the logs. Still, this might have some valid use cases. 🤷🏼‍♂️

[DavidGOrtega]

I wouldn't want

YOU wouldn't want, someone else might want to download the latest trained model epoch to test

pull data every time I want to check the logs.

In reality you would just pull latest model eventually due to the sync so you can still pull logs insanely before hitting a model download

[DavidGOrtega]

In any case I think that the flag could be transient. It we change it we can then apply and make it effective

[dmpetrov]

We also need to pull metrics (data files) when a job is running

[dmpetrov]

Actually, I'm not sure about refresh. It might be better to sync status & logs as part of refresh and sync data with another command.

@DavidGOrtega could you please clarify what do you mean by the flag? How it will look like?

[DavidGOrtega]

@dmpetrov when we create the task we can specify if this feature is enabled. Opposed to many of them that property does not destroy the task if changes.

resource "iterative_task" "tpi-examples-gpu" {
    name      = "tpi-examples-gpu"
    cloud     = "az"
    region    = "us-east"
    machine   = "Standard_NC6"
    spot      = 0
    directory = "."

    pullOnRefresh = 1

    script = ""
}