Delete task automatically with retention

[andrew-vdb]

Just like job, we should be able to delete task, preferable with some retention, deleted after x hours/x days for example, allowing us to inspect the task first before its deleted completely.

[alfpark]

You can specify a task constraint with a retention time which will automatically clean up disk space consumed by the task on the respective compute node(s). Note that this doesn't delete the task, only the associated data on the compute node(s) for the task.

It is unclear where such capability (auto deletion after retention) exists for jobs.

[andrew-vdb]

We created weekly cron job at this moment to clean up tasks. As you said, task constraint doesn't delete task. This is more feature request than question...

[chanakyaMT]

@andrew-vdb , Could you please help me with your cron script (removing secure information) to delete jobs. This will help me and others alot.

Thank you in advance.

[andrew-vdb]

public async Task CleanupCompletedTasks(int taskStorageDays = -3,int failureStorageDays = -7)
        {
            using (BatchClient batchClient = BatchClient.Open(credentials))
            {
                var listJobs = await batchClient.JobOperations.ListJobs().ToListAsync();
                foreach (var job in listJobs)
                {


                    var completedTasks = await job
                        .ListTasks(new ODATADetailLevel($"state eq 'completed' ")).ToListAsync();


                    List<Task> tasks = new List<Task>();
                    foreach (var task in completedTasks)
                    {
                        if (task.ExecutionInformation.ExitCode == 0 && task.CreationTime < DateTime.Today.AddDays(taskStorageDays))
                        {
                            
                            tasks.Add(task.DeleteAsync());
                        }
                       else if (task.CreationTime < DateTime.Today.AddDays(failureStorageDays))
                        {
                            tasks.Add(task.DeleteAsync());
                        }
                    }
                    await Task.WhenAll(tasks);
                }
            }
        }