[cinder-csi-plugin] Use Nova's os-volume_attachments API to poll attach/detach completion

[lyarwood]

/kind feature

What happened:

The current volume attach and detach code within the cinder-csi-plugin uses Nova's os-volume_attachments API [1] to initially attach [2] and later detach [3] a volume from an instance. As noted in [1] both of these APIs are async and need to be polled (with GETs against /servers/{server_id}/os-volume_attachments) after calling to determine when an operation has completed fully.

Instead in both situations cinder-csi appears to poll Cinder's volume API [4] checking the attachments listed there instead. As discussed upstream in OpenStack [5][6] there are some race conditions within Cinder at present that could allow cinder-csi to request that a volume be deleted before the original request to detach it had actually been completed. Cinder then incorrectly accepting the request to delete the volume and borking the original request to detach late in the flow.

Ultimately cinder-csi just needs to poll Nova's os-volume_attachments API to determine when an operation has completed.

[1] https://docs.openstack.org/api-ref/compute/#servers-with-volume-attachments-servers-os-volume-attachments [2] https://github.com/kubernetes/cloud-provider-openstack/blob/61f2129345c6ab854ec464bc85e0b558f011b08e/pkg/csi/cinder/openstack/openstack_volumes.go#L164-L198 [3] https://github.com/kubernetes/cloud-provider-openstack/blob/61f2129345c6ab854ec464bc85e0b558f011b08e/pkg/csi/cinder/openstack/openstack_volumes.go#L259-L287 [4] https://github.com/kubernetes/cloud-provider-openstack/blob/61f2129345c6ab854ec464bc85e0b558f011b08e/pkg/csi/cinder/openstack/openstack_volumes.go#L370-L381 [5] https://review.opendev.org/c/openstack/cinder/+/801913 [6] https://bugs.launchpad.net/cinder/+bug/1937084

What you expected to happen:

Ultimately cinder-csi just needs to poll Nova's os-volume_attachments API to determine when an operation has completed.

How to reproduce it:

I've marked this as a feature but honestly it could just be a bug so feel free to change.

Cause volumes to be attached or detached to an instance and note that Nova's os-volume_attachments API isn't used to poll the state of either operation.

Anything else we need to know?:

Environment:

• openstack-cloud-controller-manager(or other related binary) version: N/A
• OpenStack version: N/A (Nova's os-volume_attachments API has been around for all of v2)
• Others:

[ramineni]

@lyarwood Thanks for reporting the issue and detail explanation. Makes sense to me to poll nova attachments API instead . Will take a look . Thanks

[ramineni]

/assign

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ramineni]

/remove-lifecycle stale

[ramineni]

@lyarwood is this issue fixed from openstack side in latest releases?

[lyarwood]

@lyarwood is this issue fixed from openstack side in latest releases?

The Cinder change referenced above is just a bugfix resolving a race that users could hit if they incorrectly used Cinder's API to determine when a volume was attached or detached. There are likely others so this project should still move over to Nova's API to determine when an attach or detach has completed.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[chrigl]

/reopen /remove-lifecycle rotten

[k8s-ci-robot]

@chrigl: Reopened this issue.

In response to this:

/reopen /remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[seanschneeweiss]

In our environment we also observed many problems with volumes that were stuck in reserved, detaching and inconsistencies between the Nova attachments and Cinder attachments - e.g. Cinder showing available, while volume is still attached in the Nova database.

Canonical identified a bug in keystonemiddleware related to service tokens.

I found this issue when nova calls cinder with an expired X-Auth-Token but it is configured to also send a X-Service-Token. The traffic goes like this:

nova-compute -> cinder: post with X-Auth-Token and X-Service-Token cinder -> keystone: validate X-Auth-Token keystone -> cinder: returns 404 cinder -> nova-compute: returns 401 nova-compute -> cinder: retry post with new X-Service-Token cinder -> keystone: validate X-Service-Token keystone -> cinder: returns 200 showing that the token is valid cinder -> nova-compute: returns 401

As I understand Cinder should return 200 in the last message as the token is valid.

My test client is a long running service that uses the same token to communicate to nova until it receives a 401 and then generates a new one. Sometimes the token is invalidated in the middle of a transaction and nova returns 200 to the client but cinder returns 401 to nova.

The bug fix is currently being reviewed: https://review.opendev.org/c/openstack/keystonemiddleware/+/860481

As far as my understanding goes, the other bugs mentioned in this issue have already been fixed too.

_{Sean Schneeweiss [email protected], Mercedes-Benz Tech Innovation GmbH, Provider Information}

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.