CDI should resolve redirections from mirrors before handing it to nbdkit

[awels]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind enhancement

What happened: When downloading an image from a URL that is backed by a bunch of mirrors, sometimes the importer fails because one of the mirrors is misbehaving or doesn't have the file yet. This is due to nbdkit calling ranges on the URL provided, but if the URL provided is a redirection URL, it will follow it to different mirrors. If all the mirrors are working this is not a problem, however if one is misbehaving this will cause the import to fail.

What you expected to happen: I expect CDI to resolve the redirects to a real URL before passing it to nbdkit, so it doesn't do a range on many different mirrors, but instead calls the ranges on the resolved URL.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• CDI version (use kubectl get deployments cdi-deployment -o yaml):
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• Install tools:
• Others:

[cwilkers]

Reproducible (for the moment due to mirror issue) with following PVC request:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: "fedora"
  labels:
    app: containerized-data-importer
  annotations:
    cdi.kubevirt.io/storage.import.endpoint: "https://download.fedoraproject.org/pub/fedora/linux/releases/33/Cloud/x86_64/images/Fedora-Cloud-Base-33-1.2.x86_64.raw.xz"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

[kubevirt-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[awels]

/lifecycle frozen

[awels]

So the nbdkit authors have created a 'retry' plugin that will retry failed chunks. This should resolve this issue, we just need modify our code to actually use the plugin.

[awels]

https://github.com/kubevirt/containerized-data-importer/pull/2584 is merged now. This should resolve this particular issue.