Changing content checksums list results in errors and crashloopbackoff

[vkukk]

Version Please provide the versions of the pulp-operator and pulp images in use.

$ helm -n pulp list
NAME	NAMESPACE	REVISION	UPDATED                                 	STATUS  	CHART              	APP VERSION 
pulp	pulp     	1       	2024-09-03 16:54:04.919341315 +0300 EEST	deployed	pulp-operator-0.1.0	1.0.1-beta.4

apiVersion: repo-manager.pulpproject.org/v1beta2

kind: Pulp

metadata:
  name: pulp
  namespace: pulp

spec:
  allowed_content_checksums:
    - sha1
    - sha256
    - sha512

  object_storage_s3_secret: s3-secret

  database:
    # when changing pulp-postgresql env values, update secret suffix here
    external_db_secret: external-pulp-postgresql-secret-74844b4846

  cache:
    enabled: true
    external_cache_secret: external-pulp-redis-secret-d8d2dhd8bk

  api:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 250m
        memory: 256Mi
      limits:
        cpu: 1
        memory: 512Mi

  content:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 250m
        memory: 256Mi
      limits:
        cpu: 500m
        memory: 512Mi

  worker:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 500m
        memory: 500Mi
      limits:
        cpu: 2
        memory: 1Gi

Describe the bug Added section to config:

allowed_content_checksums:
    - sha1
    - sha256
    - sha512

Applied changes to Pulp deployment and all api, content and workers are crashing.

$ kubectl -n pulp logs pod/pulp-api-5bb4d46d59-7fcgx 
Waiting on postgresql to start...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 89, in _lock_storage
    asyncio.get_running_loop()
RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 58, in __getitem__
    return getattr(self._connections, alias)
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 118, in __getattr__
    return getattr(storage, key)
AttributeError: '_thread._local' object has no attribute 'default'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/wait_on_postgres.py", line 12, in <module>
    connection.ensure_connection()
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 60, in __getitem__
    if alias not in self.settings:
  File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 148, in configure_settings
    databases = super().configure_settings(databases)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 544, in <module>
    raise e
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 503, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'sha1'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.

Expected behavior Pulp would work and accept listed content checksums.

Additional context Job to update content checksums is failing

$ kubectl -n pulp logs pod/pulp-update-content-checksums-sxkkw-h86qt 
Waiting on postgresql to start...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 89, in _lock_storage
    asyncio.get_running_loop()
RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 58, in __getitem__
    return getattr(self._connections, alias)
  File "/usr/local/lib/python3.9/site-packages/asgiref/local.py", line 118, in __getattr__
    return getattr(storage, key)
AttributeError: '_thread._local' object has no attribute 'default'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/wait_on_postgres.py", line 12, in <module>
    connection.ensure_connection()
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 60, in __getitem__
    if alias not in self.settings:
  File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 148, in configure_settings
    databases = super().configure_settings(databases)
  File "/usr/local/lib/python3.9/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 544, in <module>
    raise e
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 503, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'sha1'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.
Checking for database migrations
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 416, in execute
    django.setup()
  File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/lib/python3.9/site-packages/django/apps/registry.py", line 116, in populate
    app_config.import_models()
  File "/usr/local/lib/python3.9/site-packages/django/apps/config.py", line 269, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/__init__.py", line 34, in <module>
    from .exporter import (
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/exporter.py", line 11, in <module>
    from pulpcore.app.models.repository import Repository
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/repository.py", line 34, in <module>
    from pulpcore.cache import Cache
  File "/usr/local/lib/python3.9/site-packages/pulpcore/cache/__init__.py", line 1, in <module>
    from .cache import (
  File "/usr/local/lib/python3.9/site-packages/pulpcore/cache/cache.py", line 17, in <module>
    from pulpcore.app.settings import settings
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 544, in <module>
    raise e
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/settings.py", line 503, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: There have been identified artifacts missing checksum 'sha1'. Run 'pulpcore-manager handle-artifact-checksums' first to populate missing artifact checksums.
Database migrated!
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/management/commands/handle-artifact-checksums.py", line 179, in handle
    for chunk in fp.chunks(CHUNK_SIZE):
  File "/usr/local/lib/python3.9/site-packages/django/core/files/base.py", line 55, in chunks
    self.seek(0)
  File "/usr/local/lib/python3.9/site-packages/django/core/files/utils.py", line 45, in <lambda>
    seek = property(lambda self: self.file.seek)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 48, in _get_file
    self._file = self.storage.open(self.name, "rb")
  File "/usr/local/lib/python3.9/site-packages/django/core/files/storage/base.py", line 22, in open
    return self._open(name, mode)
  File "/usr/local/lib/python3.9/site-packages/storages/backends/s3.py", line 488, in _open
    f = S3File(name, mode, self)
  File "/usr/local/lib/python3.9/site-packages/storages/backends/s3.py", line 133, in __init__
    self.obj.load(**params)
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/factory.py", line 565, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/action.py", line 88, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request