pulp-operator icon indicating copy to clipboard operation
pulp-operator copied to clipboard

[BUG] Upload/download content failures during pod rollouts

Open git-hyagi opened this issue 2 years ago • 0 comments

Describe the bug Our pods are not getting terminated in a graceful way. An image version upgrade or any change that requires deploying new pods will cause a rollout of the Deployments and the pods will be terminated even in the middle of a running task (download/upload content).

To Reproduce Steps to reproduce the behavior: Install the operator with 2 api and content replicas:

spec:
  api:
    replicas: 2
  content:
    replicas: 2
  image_version: latest
  image_web_version: latest

start to upload a file:

pulp file content upload --file Fedora-Workstation-Live-x86_64-35-1.2.iso --relative-path Fedora-Workstation-Live-x86_64-35-1.2.iso

while the upload is running, modify Pulp CR to "force" the redeploy of pods:

spec:
  image_version: stable
  image_web_version: stable

pulp client will receive the following error:

.uploads_update : put https://example-pulp.apps-crc.testing/pulp/api/v3/uploads/0188c54f-6ab8-7c75-9f6c-5014d0e0670b/
Response: 200
.uploads_update : put https://example-pulp.apps-crc.testing/pulp/api/v3/uploads/0188c54f-6ab8-7c75-9f6c-5014d0e0670b/
uploads_delete : delete https://example-pulp.apps-crc.testing/pulp/api/v3/uploads/0188c54f-6ab8-7c75-9f6c-5014d0e0670b/
Response: 204
Error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Expected behavior The pods should not be terminated if there is/are any task(s) running.

Additional context Verify which signal kubelet sends to the process in these scenarios.

git-hyagi avatar Jun 20 '23 16:06 git-hyagi