aws-janitor overly eagerly deleting IAM Role

[justinsb]

On AWS, IAM Roles have the same name (and no unique UUID). They do have a creation timestamp, however.

Our test jobs are creating IAM roles with the same name. The aws-janitor runs periodically, and if the timings work out "just so", aws-janitor will observe different IAM roles with the same name for the entire TTL window. It will then delete an IAM role, thinking that it is no longer in use, but in fact it has seen multiple different IAM roles with the same name.

I propose using CreationTimestamp to differentiate.

[olemarkus]

@justinsb is this what happens? My observation is that the janitor runs between IAM role created and IAM role used. That's also why only the pod roles are deleted, as the instance roles are used quickly after cluster creation, while pod roles are used 5-10 minutes later.

[justinsb]

There is a supplemental check that the IAM role isn't in use: https://github.com/kubernetes-sigs/boskos/blob/master/aws-janitor/resources/iam_roles.go#L89-L92

This may be what stops the instance roles from being deleted.

I do think the logic is wrong though, if we have a creationTimestamp from AWS we should treat that as the creation timestamp, not some other value that we have constructed ourselves :-)

[olemarkus]

Yeah. The instance roles are used earlier, so they get protection from that supplemental check, while the Pod ServiceAccount roles do not because they are not used yet.

I agree that if we already have a creation time check we should just use this. However, we are still seeing this after your PR merged.

https://storage.googleapis.com/kubernetes-jenkins/logs/e2e-kops-aws-k8s-latest/1556000917520650240/build-log.txt

https://storage.googleapis.com/kubernetes-jenkins/logs/e2e-kops-aws-k8s-latest/1556000917520650240/artifacts/cluster-info/kube-system/dns-controller-55fd4d7694-t4ts2/logs.txt

[olemarkus]

So we actually had a leak as well: https://github.com/kubernetes/kops/pull/14151

Let's wait for this one to merge and observe the e2es.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[olemarkus]

I think we can close this one now.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.