digdag icon indicating copy to clipboard operation
digdag copied to clipboard

Throw an error depending on the expiration of TemporalConfigStorage

Open TrsNium opened this issue 4 years ago • 1 comments

objective

KubernetesCommandExecutor now throws an exception when the execution time of a pod exceeds the defaultPodTTL. But we also need to consider that the pod execution time may exceed expiration of TemporalConfigStorage. This is because pod error detection can be done early.

problem

if we continue processing when the pod's execution time exceed the expiration of TemporalConfigStorage, the curl command will raise an error when trying to upload the archive to the expired TemporaryConfigStorage at the end of the pod process. It is inefficient to not notice error until the end of the pod process.

Fail if inTemporalConfigStorage has expired.

https://github.com/treasure-data/digdag/blob/ec9ba0e8882d6334cb271f22890716bd77be81b4/digdag-standards/src/main/java/io/digdag/standards/command/KubernetesCommandExecutor.java#L183

Fail if outTemporalConfigStorage has expired. https://github.com/treasure-data/digdag/blob/ec9ba0e8882d6334cb271f22890716bd77be81b4/digdag-standards/src/main/java/io/digdag/standards/command/KubernetesCommandExecutor.java#L213

changes

Make throwing an error depending on the expiration of TemporalConfigStorage.

reopened #1359

TrsNium avatar Dec 23 '20 05:12 TrsNium