Add ttl support for task with status `COMPLETE`

[seonWKim]

Problem description: We use cloud functions to call spring cloud dataflow tasks. Because we are executing it frequently, there are a lot of completed pods that pile up. It's bit hard to monitor which pods are being created and completed using k9s if there is some operations as above.

Solution description: I read in the documentation that we can add ttl for scheduled tasks using below property.

deployer.<application>.kubernetes.cron.ttlSecondsAfterFinished=86400

However doesn't seem to work when we pass above property when launching a single task. Will it be possible to support ttl features when launching single task as well?

Description of alternatives: Possible solution I guess is to call API that cleans up the completed tasks periodically, but I still think that using ttl would be useful because we wouldn't have to do API calls to remove completed tasks.

Additional context:

[corneil]

Have you tried adding deployer.*.spring.cloud.deployer.kubernetes.cron.ttlSecondsAfterFinished=999

[seonWKim]

@corneil I've tried, but the Completed pods were not being removed. I solved this issue by scheduling a cloud function to call tasks' cleanup API periodically.

[dhpizza]

As documentation says, this deployer.<application>.kubernetes.cron.ttlSecondsAfterFinished works on jobs only i.e. on scheduled tasks only since it uses the k8s jobs ttl controller.

Would be great to have s aimilar TTL functionality for launch tasks and streams 👍

[corneil]

@dhpizza You can configure spring.cloud.deployer.kubernetes.ttlSecondsAfterFinished with task deployment or configure it on the Data Flow Server deployment.