pulp-oci-images icon indicating copy to clipboard operation
pulp-oci-images copied to clipboard

Tracebacks in logs with quay.io/pulp/pulp

Open praiskup opened this issue 2 years ago • 11 comments

Version

Seems like container: 8dc8e64d40a1f28ee03a5546882d49214592a2a50e40203d4b87f8346301f5dc

Describe the bug Pulp doesn't seem to start correctly according to https://pulpproject.org/pulp-in-one-container/

To Reproduce Follow the steps, the following fails:

$ podman exec -it pulp bash -c 'pulpcore-manager reset-admin-password'
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "auth_user" does not exist
LINE 1: ...user"."is_active", "auth_user"."date_joined" FROM "auth_user...
                                                             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/management/commands/reset-admin-password.py", line 35, in handle
    user = User.objects.get_or_create(username="admin", is_superuser=True, is_staff=True)[0]
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 581, in get_or_create
    return self.get(**kwargs), False
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 431, in get
    num = len(clone)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "auth_user" does not exist
LINE 1: ...user"."is_active", "auth_user"."date_joined" FROM "auth_user...
                                                             ^

Additional context

podman logs pulp are 4.2MB, large to upload. The first traceback there is:

[oneshot] postgres-prepare: ^[[32mrunning Pulp migrations^[[0m
[oneshot] certs: ^[[32mfinished adding webserver certificate to the certificate store^[[0m
s6-rc: info: service certs successfully started
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py", line 393, in <module>
    with open(DB_ENCRYPTION_KEY, "rb") as key_file:
PermissionError: [Errno 13] Permission denied: '/etc/pulp/certs/database_fields.symmetric.key'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    django.setup()
  File "/usr/local/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/lib/python3.8/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/usr/local/lib/python3.8/site-packages/django/apps/config.py", line 301, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/__init__.py", line 35, in <module>
    from .exporter import (  # noqa
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/exporter.py", line 11, in <module>
    from pulpcore.app.models.repository import Repository
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/repository.py", line 26, in <module>
    from pulpcore.cache import Cache
  File "/usr/local/lib/python3.8/site-packages/pulpcore/cache/__init__.py", line 1, in <module>
    from .cache import (  # noqa
  File "/usr/local/lib/python3.8/site-packages/pulpcore/cache/cache.py", line 16, in <module>
    from pulpcore.app.settings import settings
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py", line 396, in <module>
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: Could not load DB_ENCRYPTION_KEY file '/etc/pulp/certs/database_fields.symmetric.key': [Errno 13] Permission denied: '/etc/pulp/certs/database_fields.symmetric.key'

praiskup avatar Jan 09 '23 09:01 praiskup

What's the podman version? where are you running it? https://pulpproject.org/pulp-in-one-container/#podman-on-centos-7

ipanova avatar Jan 09 '23 13:01 ipanova

14:40:50 ~$ podman --version
podman version 4.3.1
14:40:53 ~$ rpm -q podman
podman-4.3.1-1.fc37.x86_64

praiskup avatar Jan 09 '23 13:01 praiskup

Also note that I'm not quite sure I'm using the right image.

praiskup avatar Jan 09 '23 14:01 praiskup

I tried to reproduce using the latest image from quay.io and I could not reproduce it.

quay.io/pulp/pulp            latest      728f8846e816  2 weeks ago  2.18 GB

It seems like you have a permissions problem with the /etc/pulp/certs/ directory ... Here is what the permissions look like for me:

[dkliban@fedora ~]$ podman exec -it pulp bash
[root@25aedd4dc552 /]# ls -la /etc/pulp/
total 4
drwxr-xr-x. 1 root root  32 Jan  9 20:36 .
drwxr-xr-x. 1 root root   6 Dec 23 01:32 ..
drwxr-xr-x. 1 root root 248 Jan  9 20:36 certs
-rw-r--r--. 1 root root 162 Jan  9 20:35 settings.py
[root@25aedd4dc552 /]# ls -la /etc/pulp/certs/
total 24
drwxr-xr-x. 1 root root  248 Jan  9 20:36 .
drwxr-xr-x. 1 root root   32 Jan  9 20:36 ..
-rw-r-----. 1 root pulp   45 Jan  9 20:36 database_fields.symmetric.key
-rw-r--r--. 1 root pulp 1334 Jan  9 20:36 pulp_webserver.crt
-rw-r--r--. 1 root pulp 1175 Jan  9 20:36 pulp_webserver.csr
-rw-------. 1 root pulp 1675 Jan  9 20:36 pulp_webserver.key
-rw-------. 1 pulp pulp  227 Jan  9 20:36 token_private_key.pem
-rw-r--r--. 1 pulp pulp  178 Jan  9 20:36 token_public_key.pem

Can you share the output of the above commands for you?

dkliban avatar Jan 09 '23 20:01 dkliban

Looks promising, thanks!

[root@21e381ccf81f /]# ls -la /etc/pulp/
total 12
drwx------.  3 root root   80 Jan 10 09:19 .
drwxr-xr-x. 72 root root 4096 Dec 23 01:32 ..
drwxr-xr-x.  2 root root  160 Jan 10 09:19 certs
-rw-------.  1 root root  165 Jan 10 09:18 settings.py

praiskup avatar Jan 10 '23 09:01 praiskup

Indeed, I temporarily switched umask from 0077 to 0022, and recreated from scratch. That helped.

Could the permissions be fixed automatically? I can see some tasks like:

fix-attrs: warning: fix-attrs is deprecated, please fix volume permissions in your container manager instead
fix-attrs: warning: some fix files failed to apply

I'm actually in a user namespace; so the files in /etc/pulp in container are owned by "fake root" (== my non-privileged praiskup user). We should have the rights to fix the permissions bits.

Nevertheless, it eventually worked! Thank you.

For the sake of completeness, podman logs pulp shows:

Calling /etc/init/postgres-prepare
2023-01-10 09:29:25.952 UTC [12591] LOG:  redirecting log output to logging collector process
2023-01-10 09:29:25.952 UTC [12591] HINT:  Future log output will appear in directory "log".
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 363, in execute
    settings.INSTALLED_APPS
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py", line 299, in <module>
    from dynaconf import DjangoDynaconf, Validator  # noqa
  File "/usr/local/lib/python3.8/site-packages/dynaconf/__init__.py", line 3, in <module>
    from dynaconf.base import LazySettings  # noqa
  File "/usr/local/lib/python3.8/site-packages/dynaconf/base.py", line 17, in <module>
    from dynaconf.loaders import env_loader
  File "/usr/local/lib/python3.8/site-packages/dynaconf/loaders/env_loader.py", line 8, in <module>
    from dynaconf.vendor.dotenv import cli as dotenv_cli
  File "/usr/local/lib/python3.8/site-packages/dynaconf/vendor/dotenv/cli.py", line 16, in <module>
    @click.option('-f','--file',default=os.path.join(os.getcwd(),'.env'),type=click.Path(exists=_A),help='Location of the .env file, defaults to .env file in current working directory.')
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 363, in execute
    settings.INSTALLED_APPS
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py", line 299, in <module>
    from dynaconf import DjangoDynaconf, Validator  # noqa
  File "/usr/local/lib/python3.8/site-packages/dynaconf/__init__.py", line 3, in <module>
    from dynaconf.base import LazySettings  # noqa
  File "/usr/local/lib/python3.8/site-packages/dynaconf/base.py", line 17, in <module>
    from dynaconf.loaders import env_loader
  File "/usr/local/lib/python3.8/site-packages/dynaconf/loaders/env_loader.py", line 8, in <module>
    from dynaconf.vendor.dotenv import cli as dotenv_cli
  File "/usr/local/lib/python3.8/site-packages/dynaconf/vendor/dotenv/cli.py", line 16, in <module>
    @click.option('-f','--file',default=os.path.join(os.getcwd(),'.env'),type=click.Path(exists=_A),help='Location of the .env file, defaults to .env file in current working directory.')
FileNotFoundError: [Errno 2] No such file or directory
Database migrated!
ready
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 363, in execute
    settings.INSTALLED_APPS
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py", line 299, in <module>
    from dynaconf import DjangoDynaconf, Validator  # noqa
  File "/usr/local/lib/python3.8/site-packages/dynaconf/__init__.py", line 3, in <module>
    from dynaconf.base import LazySettings  # noqa
  File "/usr/local/lib/python3.8/site-packages/dynaconf/base.py", line 17, in <module>
    from dynaconf.loaders import env_loader
  File "/usr/local/lib/python3.8/site-packages/dynaconf/loaders/env_loader.py", line 8, in <module>
    from dynaconf.vendor.dotenv import cli as dotenv_cli
  File "/usr/local/lib/python3.8/site-packages/dynaconf/vendor/dotenv/cli.py", line 16, in <module>
    @click.option('-f','--file',default=os.path.join(os.getcwd(),'.env'),type=click.Path(exists=_A),help='Location of the .env file, defaults to .env file in current working directory.')
FileNotFoundError: [Errno 2] No such file or directory
Database migrated!
ready
Database migrated!
ready

praiskup avatar Jan 10 '23 09:01 praiskup

Also:

[2023-01-10 09:42:18 +0000] [12692] [ERROR] Exception in gunicorn worker
Traceback (most recent call last):
  File "/usr/local/lib64/python3.8/site-packages/aiohttp/worker.py", line 57, in run
    self.loop.run_until_complete(self._task)
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib64/python3.8/site-packages/aiohttp/worker.py", line 70, in _run
    wsgi = await self.wsgi()
  File "/usr/local/lib/python3.8/site-packages/pulpcore/content/__init__.py", line 65, in server
    os.chdir(settings.WORKING_DIRECTORY)
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/tmp'

praiskup avatar Jan 10 '23 09:01 praiskup

Debian GNU/Linux 11.6 (bullseye) :heavy_check_mark:
  1. podman version 3.0.1
  2. Python 3.9.2
  3. pip 22.3.1 from /home/debian/.local/lib/python3.9/site-packages/pip (python 3.9)
  4. pulp3 command line interface, version 0.16.0

Step 1 - Without SELinux

  podman run --detach \
               --publish 8080:80 \
               --name pulp \
               --volume "$(pwd)/settings":/etc/pulp \
               --volume "$(pwd)/pulp_storage":/var/lib/pulp \
               --volume "$(pwd)/pgsql":/var/lib/pgsql \
               --volume "$(pwd)/containers":/var/lib/containers \
               --device /dev/fuse \
               quay.io/pulp/pulp
  Trying to pull quay.io/pulp/pulp:latest...
  Getting image source signatures
  Copying blob 26c922825726 done  
  Copying blob c85324ce8e2d done  
  Copying blob 0d557d32f54e done  
  Copying blob f1ee40d9db4a done  
  Copying blob 17facd475902 done  
  Copying blob 6c5de04c936d done  
  Copying blob a86af3d96bf9 done  
  Copying blob b6b91d522a42 done  
  Copying blob 59e45885fb1c done 
  # ...

Step 2

   curl debian-test-pulp:8080/pulp/api/v3/status/ -vv | jq 
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.240.0.48:8080...
  * Connected to debian-test-pulp (10.240.0.48) port 8080 (#0)
  > GET /pulp/api/v3/status/ HTTP/1.1
  > Host: debian-test-pulp:8080
  > User-Agent: curl/7.74.0
  > Accept: */*
  > 
  * Mark bundle as not supporting multiuse
  < HTTP/1.1 200 OK
  < Server: nginx/1.14.1
  < Date: Sat, 14 Jan 2023 11:14:42 GMT
  < Content-Type: application/json
  < Content-Length: 1426
  < Connection: keep-alive
  < Vary: Accept
  < Allow: GET, HEAD, OPTIONS
  < X-Frame-Options: DENY
  < X-Content-Type-Options: nosniff
  < Referrer-Policy: same-origin
  < Correlation-ID: 023194d247a841079453c545fd202fde
  < Access-Control-Expose-Headers: Correlation-ID
  < 
  { [1426 bytes data]
  100  1426  100  1426    0     0   3197      0 --:--:-- --:--:-- --:--:--  3190
  * Connection #0 to host debian-test-pulp left intact
  {
    "versions": [
      {
        "component": "core",
        "version": "3.22.0",
        "package": "pulpcore"
      },
      {
        "component": "rpm",
        "version": "3.18.9",
        "package": "pulp-rpm"
      },
      {
        "component": "python",
        "version": "3.8.0",
        "package": "pulp-python"
      },
      {
        "component": "deb",
        "version": "2.20.1",
        "package": "pulp_deb"
      },
      {
        "component": "container",
        "version": "2.14.3",
        "package": "pulp-container"
      },
      {
        "component": "file",
        "version": "1.11.2",
        "package": "pulp-file"
      },
      {
        "component": "certguard",
        "version": "1.5.5",
        "package": "pulp-certguard"
      },
      {
        "component": "ansible",
        "version": "0.16.0",
        "package": "pulp-ansible"
      },
      {
        "component": "maven",
        "version": "0.3.3",
        "package": "pulp-maven"
      }
    ],
    "online_workers": [
      {
        "pulp_href": "/pulp/api/v3/workers/86c5003e-f2f0-43cd-b73c-7642f3ff3ab4/",
        "pulp_created": "2023-01-14T11:14:23.795433Z",
        "name": "12652@56224fb13195",
        "last_heartbeat": "2023-01-14T11:14:33.837858Z",
        "current_task": null
      },
      {
        "pulp_href": "/pulp/api/v3/workers/a43ae658-4b50-4b4a-bc0c-da18f803e161/",
        "pulp_created": "2023-01-14T11:14:24.344275Z",
        "name": "12692@56224fb13195",
        "last_heartbeat": "2023-01-14T11:14:35.267775Z",
        "current_task": null
      }
    ],
    "online_content_apps": [
      {
        "name": "12723@56224fb13195",
        "last_heartbeat": "2023-01-14T11:14:36.306386Z"
      },
      {
        "name": "12722@56224fb13195",
        "last_heartbeat": "2023-01-14T11:14:36.309315Z"
      }
    ],
    "database_connection": {
      "connected": true
    },
    "redis_connection": {
      "connected": false
    },
    "storage": null,
    "content_settings": {
      "content_origin": "http://debian-test-pulp:8080",
      "content_path_prefix": "/pulp/content/"
    }
  }

Step 3

  pip install pulp-cli[pygments]
  Defaulting to user installation because normal site-packages is not writeable
  Requirement already satisfied: pulp-cli[pygments] in ./.local/lib/python3.9/site-packages (0.16.0)
  Requirement already satisfied: requests~=2.24 in ./.local/lib/python3.9/site-packages (from pulp-cli[pygments]) (2.28.2)
  Requirement already satisfied: click<9.0.0,>=8.0.0 in ./.local/lib/python3.9/site-packages (from pulp-cli[pygments]) (8.1.3)
  Requirement already satisfied: toml==0.10.2 in ./.local/lib/python3.9/site-packages (from pulp-cli[pygments]) (0.10.2)
  Requirement already satisfied: PyYAML~=5.3 in ./.local/lib/python3.9/site-packages (from pulp-cli[pygments]) (5.4.1)
  Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from pulp-cli[pygments]) (52.0.0)
  Requirement already satisfied: schema==0.7.5 in ./.local/lib/python3.9/site-packages (from pulp-cli[pygments]) (0.7.5)
  Requirement already satisfied: packaging in ./.local/lib/python3.9/site-packages (from pulp-cli[pygments]) (23.0)
  Requirement already satisfied: pygments in ./.local/lib/python3.9/site-packages (from pulp-cli[pygments]) (2.14.0)
  Requirement already satisfied: contextlib2>=0.5.5 in ./.local/lib/python3.9/site-packages (from schema==0.7.5->pulp-cli[pygments]) (21.6.0)
  Requirement already satisfied: certifi>=2017.4.17 in ./.local/lib/python3.9/site-packages (from requests~=2.24->pulp-cli[pygments]) (2022.12.7)
  Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.local/lib/python3.9/site-packages (from requests~=2.24->pulp-cli[pygments]) (1.26.14)
  Requirement already satisfied: charset-normalizer<4,>=2 in ./.local/lib/python3.9/site-packages (from requests~=2.24->pulp-cli[pygments]) (3.0.1)
  Requirement already satisfied: idna<4,>=2.5 in ./.local/lib/python3.9/site-packages (from requests~=2.24->pulp-cli[pygments]) (3.4)

Note that it is better to use sudo to install the client for Debian GNU/Linux systems, otherwise you will have the following WARNING:

   WARNING: The script pulp is installed in '/home/user/.local/bin' which is not on PATH.
    Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  Successfully installed pulp-cli-0.16.0

and as a result:

  pulp config create --username admin --base-url http://localhost:8080 --password test12345
  -bash: pulp: command not found

Because :

echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

So, now you have at least two available options to solve this.

  1. Simply by installing the client on your entire system (not just for your local user):
    sudo pip install pulp-cli[pygments]
    
  2. Fixing the $PATH:
    echo $PATH
    /home/mylocaluser/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
    
    export PATH="/home/$USER/.local/bin:$PATH"
    
    echo $PATH
    /home/mylocaluser/.local/bin:/home/mylocaluser/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
        ```
    
    

Step 4

pulp config create --username admin --base-url http://localhost:8080 --password test12345
Created config file at '/home/debian/.config/pulp/cli.toml'.
cat /home/debian/.config/pulp/cli.toml
[cli]
username = "admin"
base_url = "http://localhost:8080"
password = "test12345"
api_root = "/pulp/"
cert = ""
key = ""
verify_ssl = true
format = "json"
dry_run = false
timeout = 0
verbose = 0
Ubuntu 22.04.1 LTS Minimal :heavy_check_mark:
  1. podman version 3.4.4
  2. Python 3.10.6
  3. pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
  4. pulp3 command line interface, version 0.16.0

Step 1 - Without SELinux

podman run --detach \
           --publish 8080:80 \
           --name pulp \
           --volume "$(pwd)/settings":/etc/pulp \
           --volume "$(pwd)/pulp_storage":/var/lib/pulp \
           --volume "$(pwd)/pgsql":/var/lib/pgsql \
           --volume "$(pwd)/containers":/var/lib/containers \
           --device /dev/fuse \
           quay.io/pulp/pulp
# ...
Copying blob 84c72a4efabe done  
Copying config 1ae91f80dd done  
Writing manifest to image destination
Storing signatures
5a51e962351ec33688955749327d55ad8e7c665b018e1a41b7069817973db03f

Step 2

   podman exec -it pulp bash -c 'pulpcore-manager reset-admin-password'
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py", line 393, in <module>
  with open(DB_ENCRYPTION_KEY, "rb") as key_file:
FileNotFoundError: [Errno 2] No such file or directory: '/etc/pulp/certs/database_fields.symmetric.key'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/pulpcore-manager", line 8, in <module>
  sys.exit(manage())
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
  execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
  utility.execute()
File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
  django.setup()
File "/usr/local/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
  apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.8/site-packages/django/apps/registry.py", line 114, in populate
  app_config.import_models()
File "/usr/local/lib/python3.8/site-packages/django/apps/config.py", line 301, in import_models
  self.models_module = import_module(models_module_name)
File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
  return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/__init__.py", line 35, in <module>
  from .exporter import (  # noqa
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/exporter.py", line 11, in <module>
  from pulpcore.app.models.repository import Repository
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/repository.py", line 26, in <module>
  from pulpcore.cache import Cache
File "/usr/local/lib/python3.8/site-packages/pulpcore/cache/__init__.py", line 1, in <module>
  from .cache import (  # noqa
File "/usr/local/lib/python3.8/site-packages/pulpcore/cache/cache.py", line 16, in <module>
  from pulpcore.app.settings import settings
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py", line 396, in <module>
  raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: Could not load DB_ENCRYPTION_KEY file '/etc/pulp/certs/database_fields.symmetric.key': [Errno 2] No such file or directory: '/etc/pulp/certs/database_fields.symmetric.key'
   curl test-pulp-ubuntu:8080/pulp/api/v3/status/ -v | jq .
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.240.0.49:8080...
* Connected to test-pulp-ubuntu (10.240.0.49) port 8080 (#0)
> GET /pulp/api/v3/status/ HTTP/1.1
> Host: test-pulp-ubuntu:8080
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.14.1
< Date: Sat, 14 Jan 2023 12:09:30 GMT
< Content-Type: application/json
< Content-Length: 1433
< Connection: keep-alive
< Vary: Accept
< Allow: GET, HEAD, OPTIONS
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< Referrer-Policy: same-origin
< Correlation-ID: fd6dd2e38c02467cbacd4c4b89d4d43c
< Access-Control-Expose-Headers: Correlation-ID
< 
{ [1433 bytes data]
100  1433  100  1433    0     0   2864      0 --:--:-- --:--:-- --:--:--  2866
* Connection #0 to host test-pulp-ubuntu left intact
{
"versions": [
  {
    "component": "core",
    "version": "3.22.0",
    "package": "pulpcore"
  },
  {
    "component": "rpm",
    "version": "3.18.9",
    "package": "pulp-rpm"
  },
  {
    "component": "python",
    "version": "3.8.0",
    "package": "pulp-python"
  },
  {
    "component": "deb",
    "version": "2.20.1",
    "package": "pulp_deb"
  },
  {
    "component": "container",
    "version": "2.14.3",
    "package": "pulp-container"
  },
  {
    "component": "file",
    "version": "1.11.2",
    "package": "pulp-file"
  },
  {
    "component": "certguard",
    "version": "1.5.5",
    "package": "pulp-certguard"
  },
  {
    "component": "ansible",
    "version": "0.16.0",
    "package": "pulp-ansible"
  },
  {
    "component": "maven",
    "version": "0.3.3",
    "package": "pulp-maven"
  }
],
"online_workers": [
  {
    "pulp_href": "/pulp/api/v3/workers/b84c980d-011a-4825-aa13-ca18c7bf9b66/",
    "pulp_created": "2023-01-14T12:08:06.712041Z",
    "name": "12647@5a51e962351e",
    "last_heartbeat": "2023-01-14T12:09:27.066489Z",
    "current_task": null
  },
  {
    "pulp_href": "/pulp/api/v3/workers/187cb257-24fc-4076-8ad5-c072ad7f4895/",
    "pulp_created": "2023-01-14T12:08:07.726668Z",
    "name": "12687@5a51e962351e",
    "last_heartbeat": "2023-01-14T12:09:28.290061Z",
    "current_task": null
  }
],
"online_content_apps": [
  {
    "name": "12697@5a51e962351e",
    "last_heartbeat": "2023-01-14T12:09:23.301940Z"
  },
  {
    "name": "12698@5a51e962351e",
    "last_heartbeat": "2023-01-14T12:09:23.487559Z"
  }
],
"database_connection": {
  "connected": true
},
"redis_connection": {
  "connected": false
},
"storage": null,
"content_settings": {
  "content_origin": "http://test-pulp-ubuntu:8080",
  "content_path_prefix": "/pulp/content/"
}
}
podman exec -it pulp bash -c 'pulpcore-manager reset-admin-password'
Please enter new password for user "admin": 
Please enter new password for user "admin" again: 
Successfully set password for "admin" user.     
podman container ls
CONTAINER ID  IMAGE                     COMMAND     CREATED        STATUS            PORTS                 NAMES
5a51e962351e  quay.io/pulp/pulp:latest  /init       6 minutes ago  Up 6 minutes ago  0.0.0.0:8080->80/tcp  pul

Step 3

sudo pip install pulp-cli[pygments]
Collecting pulp-cli[pygments]
Downloading pulp_cli-0.16.0-py3-none-any.whl (121 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.3/121.3 KB 3.1 MB/s eta 0:00:00
Collecting packaging
Downloading packaging-23.0-py3-none-any.whl (42 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.7/42.7 KB 7.1 MB/s eta 0:00:00
Collecting schema==0.7.5
Downloading schema-0.7.5-py2.py3-none-any.whl (17 kB)
Collecting click<9.0.0,>=8.0.0
Downloading click-8.1.3-py3-none-any.whl (96 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.6/96.6 KB 14.1 MB/s eta 0:00:00
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from pulp-cli[pygments]) (59.6.0)
Requirement already satisfied: requests~=2.24 in /usr/lib/python3/dist-packages (from pulp-cli[pygments]) (2.25.1)
Requirement already satisfied: PyYAML~=5.3 in /usr/lib/python3/dist-packages (from pulp-cli[pygments]) (5.4.1)
Collecting toml==0.10.2
Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting pygments
Downloading Pygments-2.14.0-py3-none-any.whl (1.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 30.7 MB/s eta 0:00:00
Collecting contextlib2>=0.5.5
Downloading contextlib2-21.6.0-py2.py3-none-any.whl (13 kB)
Installing collected packages: toml, pygments, packaging, contextlib2, click, schema, pulp-cli
Successfully installed click-8.1.3 contextlib2-21.6.0 packaging-23.0 pulp-cli-0.16.0 pygments-2.14.0 schema-0.7.5 toml-0.10.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

Step 4

pulp config create --username admin --base-url http://localhost:8080 --password test12345
Created config file at '/home/ubuntu/.config/pulp/cli.toml'.
cat /home/ubuntu/.config/pulp/cli.toml
[cli]
username = "admin"
base_url = "http://localhost:8080"
password = "test12345"
api_root = "/pulp/"
cert = ""
key = ""
verify_ssl = true
format = "json"
dry_run = false
timeout = 0
verbose = 0
Fedora Linux 36 (Cloud Edition) :heavy_check_mark:
  1. podman version 4.3.1
  2. Python 3.10.9
  3. pip 22.3.1 from /home/fedora/.local/lib/python3.10/site-packages/pip (python 3.10)
  4. pulp3 command line interface, version 0.16.0

Step 1 - With SELinux

cat settings/settings.py 
CONTENT_ORIGIN='http://test-pulp-fedora.c.openshift-gce-devel.internal:8080'
ANSIBLE_API_HOSTNAME='http://test-pulp-fedora.c.openshift-gce-devel.internal:8080'
ANSIBLE_CONTENT_HOSTNAME='http://test-pulp-fedora.c.openshift-gce-devel.internal:8080/pulp/content'
TOKEN_AUTH_DISABLED=True
podman run --detach \
           --publish 8080:80 \
           --name pulp \
           --volume "$(pwd)/settings":/etc/pulp:Z \
           --volume "$(pwd)/pulp_storage":/var/lib/pulp:Z \
           --volume "$(pwd)/pgsql":/var/lib/pgsql:Z \
           --volume "$(pwd)/containers":/var/lib/containers:Z \
           --device /dev/fuse \
           quay.io/pulp/pulp
# ...
Copying blob 8fa5ec848b2f done  
Copying blob 84c72a4efabe done  
Copying blob 873bb322dac3 done  
Copying config 1ae91f80dd done  
Writing manifest to image destination
Storing signatures
c4ca27d501648cb13323f3a09879115aaabfcf6b308a0fa607d3614c6377fd1e

Step 2

podman exec -it pulp bash -c 'pulpcore-manager reset-admin-password'
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py", line 393, in <module>
  with open(DB_ENCRYPTION_KEY, "rb") as key_file:
FileNotFoundError: [Errno 2] No such file or directory: '/etc/pulp/certs/database_fields.symmetric.key'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/pulpcore-manager", line 8, in <module>
  sys.exit(manage())
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
  execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
  utility.execute()
File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
  django.setup()
File "/usr/local/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
  apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.8/site-packages/django/apps/registry.py", line 114, in populate
  app_config.import_models()
File "/usr/local/lib/python3.8/site-packages/django/apps/config.py", line 301, in import_models
  self.models_module = import_module(models_module_name)
File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
  return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/__init__.py", line 35, in <module>
  from .exporter import (  # noqa
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/exporter.py", line 11, in <module>
  from pulpcore.app.models.repository import Repository
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/repository.py", line 26, in <module>
  from pulpcore.cache import Cache
File "/usr/local/lib/python3.8/site-packages/pulpcore/cache/__init__.py", line 1, in <module>
  from .cache import (  # noqa
File "/usr/local/lib/python3.8/site-packages/pulpcore/cache/cache.py", line 16, in <module>
  from pulpcore.app.settings import settings
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py", line 396, in <module>
  raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: Could not load DB_ENCRYPTION_KEY file '/etc/pulp/certs/database_fields.symmetric.key': [Errno 2] No such file or directory: '/etc/pulp/certs/database_fields.symmetric.key'
podman container ls
CONTAINER ID  IMAGE                     COMMAND     CREATED         STATUS             PORTS                 NAMES
c4ca27d50164  quay.io/pulp/pulp:latest  /init       36 seconds ago  Up 36 seconds ago  0.0.0.0:8080->80/tcp  pulp
podman exec -it pulp bash -c 'pulpcore-manager reset-admin-password'
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py", line 393, in <module>
  with open(DB_ENCRYPTION_KEY, "rb") as key_file:
FileNotFoundError: [Errno 2] No such file or directory: '/etc/pulp/certs/database_fields.symmetric.key'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/pulpcore-manager", line 8, in <module>
  sys.exit(manage())
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
  execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
  utility.execute()
File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
  django.setup()
File "/usr/local/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
  apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.8/site-packages/django/apps/registry.py", line 114, in populate
  app_config.import_models()
File "/usr/local/lib/python3.8/site-packages/django/apps/config.py", line 301, in import_models
  self.models_module = import_module(models_module_name)
File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
  return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/__init__.py", line 35, in <module>
  from .exporter import (  # noqa
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/exporter.py", line 11, in <module>
  from pulpcore.app.models.repository import Repository
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/repository.py", line 26, in <module>
  from pulpcore.cache import Cache
File "/usr/local/lib/python3.8/site-packages/pulpcore/cache/__init__.py", line 1, in <module>
  from .cache import (  # noqa
File "/usr/local/lib/python3.8/site-packages/pulpcore/cache/cache.py", line 16, in <module>
  from pulpcore.app.settings import settings
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py", line 396, in <module>
  raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: Could not load DB_ENCRYPTION_KEY file '/etc/pulp/certs/database_fields.symmetric.key': [Errno 2] No such file or directory: '/etc/pulp/certs/database_fields.symmetric.key'
podman container ls
CONTAINER ID  IMAGE                     COMMAND     CREATED             STATUS                 PORTS                 NAMES
c4ca27d50164  quay.io/pulp/pulp:latest  /init       About a minute ago  Up About a minute ago  0.0.0.0:8080->80/tcp  pulp
podman exec -it pulp bash -c 'pulpcore-manager reset-admin-password'
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
  return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "auth_user" does not exist
LINE 1: ...user"."is_active", "auth_user"."date_joined" FROM "auth_user...
                                                           ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/bin/pulpcore-manager", line 8, in <module>
  sys.exit(manage())
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
  execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
  utility.execute()
File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
  self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
  self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
  output = self.handle(*args, **options)
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/management/commands/reset-admin-password.py", line 35, in handle
  user = User.objects.get_or_create(username="admin", is_superuser=True, is_staff=True)[0]
File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
  return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 581, in get_or_create
  return self.get(**kwargs), False
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 431, in get
  num = len(clone)
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 262, in __len__
  self._fetch_all()
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all
  self._result_cache = list(self._iterable_class(self))
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 51, in __iter__
  results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
  cursor.execute(sql, params)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
  return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
  return executor(sql, params, many, context)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
  return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
  raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
  return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "auth_user" does not exist
LINE 1: ...user"."is_active", "auth_user"."date_joined" FROM "auth_user...
podman container ls
CONTAINER ID  IMAGE                     COMMAND     CREATED        STATUS            PORTS                 NAMES
c4ca27d50164  quay.io/pulp/pulp:latest  /init       2 minutes ago  Up 2 minutes ago  0.0.0.0:8080->80/tcp  pulp
podman exec -it pulp bash -c 'pulpcore-manager reset-admin-password'
Please enter new password for user "admin": 
Please enter new password for user "admin" again: 
Successfully set password for "admin" user.
podman container ls
CONTAINER ID  IMAGE                     COMMAND     CREATED        STATUS            PORTS                 NAMES
c4ca27d50164  quay.io/pulp/pulp:latest  /init       2 minutes ago  Up 3 minutes ago  0.0.0.0:8080->80/tcp  pulp
curl localhost:8080/pulp/api/v3/status/ | jq
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
100  1464  100  1464    0     0   2926      0 --:--:-- --:--:-- --:--:--  2945
{
"versions": [
  {
    "component": "core",
    "version": "3.22.0",
    "package": "pulpcore"
  },
  {
    "component": "rpm",
    "version": "3.18.9",
    "package": "pulp-rpm"
  },
  {
    "component": "python",
    "version": "3.8.0",
    "package": "pulp-python"
  },
  {
    "component": "deb",
    "version": "2.20.1",
    "package": "pulp_deb"
  },
  {
    "component": "container",
    "version": "2.14.3",
    "package": "pulp-container"
  },
  {
    "component": "file",
    "version": "1.11.2",
    "package": "pulp-file"
  },
  {
    "component": "certguard",
    "version": "1.5.5",
    "package": "pulp-certguard"
  },
  {
    "component": "ansible",
    "version": "0.16.0",
    "package": "pulp-ansible"
  },
  {
    "component": "maven",
    "version": "0.3.3",
    "package": "pulp-maven"
  }
],
"online_workers": [
  {
    "pulp_href": "/pulp/api/v3/workers/3494353d-5d8f-4c23-821e-c4dae6cf494d/",
    "pulp_created": "2023-01-14T12:39:45.263615Z",
    "name": "12662@c4ca27d50164",
    "last_heartbeat": "2023-01-14T12:39:45.269086Z",
    "current_task": null
  },
  {
    "pulp_href": "/pulp/api/v3/workers/3bae2bdb-832a-4c90-ab1b-1334f12ec6ee/",
    "pulp_created": "2023-01-14T12:39:45.013684Z",
    "name": "12695@c4ca27d50164",
    "last_heartbeat": "2023-01-14T12:39:55.176903Z",
    "current_task": null
  }
],
"online_content_apps": [
  {
    "name": "12710@c4ca27d50164",
    "last_heartbeat": "2023-01-14T12:39:57.990674Z"
  },
  {
    "name": "12711@c4ca27d50164",
    "last_heartbeat": "2023-01-14T12:39:58.438233Z"
  }
],
"database_connection": {
  "connected": true
},
"redis_connection": {
  "connected": false
},
"storage": null,
"content_settings": {
  "content_origin": "http://test-pulp-fedora.c.openshift-gce-devel.internal:8080",
  "content_path_prefix": "/pulp/content/"
}
}
podman exec -it pulp bash -c 'pulpcore-manager reset-admin-password'
Please enter new password for user "admin": 
Please enter new password for user "admin" again: 
Successfully set password for "admin" user.     
podman container ls
CONTAINER ID  IMAGE                     COMMAND     CREATED        STATUS            PORTS                 NAMES
5a51e962351e  quay.io/pulp/pulp:latest  /init       6 minutes ago  Up 6 minutes ago  0.0.0.0:8080->80/tcp  pul

Step 3

Note: On Fedora pip seems to work differently. You do not need to install pulp-cli[pygments] with sudo.

pip install pulp-cli[pygments]
Defaulting to user installation because normal site-packages is not writeable
Collecting pulp-cli[pygments]
Downloading pulp_cli-0.16.0-py3-none-any.whl (121 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.3/121.3 kB 3.8 MB/s eta 0:00:00
Collecting packaging
Downloading packaging-23.0-py3-none-any.whl (42 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.7/42.7 kB 6.3 MB/s eta 0:00:00
Collecting click<9.0.0,>=8.0.0
Downloading click-8.1.3-py3-none-any.whl (96 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.6/96.6 kB 14.0 MB/s eta 0:00:00
Requirement already satisfied: setuptools in /usr/lib/python3.10/site-packages (from pulp-cli[pygments]) (59.6.0)
Collecting schema==0.7.5
Downloading schema-0.7.5-py2.py3-none-any.whl (17 kB)
Collecting PyYAML~=5.3
Downloading PyYAML-5.4.1.tar.gz (175 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 kB 23.0 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting requests~=2.24
Downloading requests-2.28.2-py3-none-any.whl (62 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 8.6 MB/s eta 0:00:00
Collecting toml==0.10.2
Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting pygments
Downloading Pygments-2.14.0-py3-none-any.whl (1.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 34.6 MB/s eta 0:00:00
Collecting contextlib2>=0.5.5
Downloading contextlib2-21.6.0-py2.py3-none-any.whl (13 kB)
Collecting urllib3<1.27,>=1.21.1
Downloading urllib3-1.26.14-py2.py3-none-any.whl (140 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.6/140.6 kB 19.0 MB/s eta 0:00:00
Collecting idna<4,>=2.5
Downloading idna-3.4-py3-none-any.whl (61 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 7.9 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 19.1 MB/s eta 0:00:00
Collecting charset-normalizer<4,>=2
Downloading charset_normalizer-3.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (198 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 198.8/198.8 kB 24.5 MB/s eta 0:00:00
Building wheels for collected packages: PyYAML
Building wheel for PyYAML (pyproject.toml) ... done
Created wheel for PyYAML: filename=PyYAML-5.4.1-cp310-cp310-linux_x86_64.whl size=45658 sha256=6008d3d40ad8d48753a49bfbf66f903c01df75655f5867304f8b26e710bf8028
Stored in directory: /home/fedora/.cache/pip/wheels/10/3f/c1/6feec52270653969575b83de934b04be56d7a9e961be5041ea
Successfully built PyYAML
Installing collected packages: charset-normalizer, urllib3, toml, PyYAML, pygments, packaging, idna, contextlib2, click, certifi, schema, requests, pulp-cli
Successfully installed PyYAML-5.4.1 certifi-2022.12.7 charset-normalizer-3.0.1 click-8.1.3 contextlib2-21.6.0 idna-3.4 packaging-23.0 pulp-cli-0.16.0 pygments-2.14.0 requests-2.28.2 schema-0.7.5 toml-0.10.2 urllib3-1.26.14
echo $PATH
/home/fedora/.local/bin:/home/fedora/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin

Step 4

pulp config create --username admin --base-url http://localhost:8080 --password test12345
Created config file at '/home/fedora/.config/pulp/cli.toml'.
cat /home/fedora/.config/pulp/cli.toml
[cli]
username = "admin"
base_url = "http://localhost:8080"
password = "test12345"
api_root = "/pulp/"
cert = ""
key = ""
verify_ssl = true
format = "json"
dry_run = false
timeout = 0
verbose = 0
pulp status
{
  "versions": [
    {
      "component": "core",
      "version": "3.22.0",
      "package": "pulpcore"
    },
    {
      "component": "rpm",
      "version": "3.18.9",
      "package": "pulp-rpm"
    },
    {
      "component": "python",
      "version": "3.8.0",
      "package": "pulp-python"
    },
    {
      "component": "deb",
      "version": "2.20.1",
      "package": "pulp_deb"
    },
    {
      "component": "container",
      "version": "2.14.3",
      "package": "pulp-container"
    },
    {
      "component": "file",
      "version": "1.11.2",
      "package": "pulp-file"
    },
    {
      "component": "certguard",
      "version": "1.5.5",
      "package": "pulp-certguard"
    },
    {
      "component": "ansible",
      "version": "0.16.0",
      "package": "pulp-ansible"
    },
    {
      "component": "maven",
      "version": "0.3.3",
      "package": "pulp-maven"
    }
  ],
  "online_workers": [
    {
      "pulp_href": "/pulp/api/v3/workers/45d38632-9136-42ab-ac55-87cf17eed086/",
      "pulp_created": "2023-01-14T12:54:44.920187Z",
      "name": "392@1bc295b1c711",
      "last_heartbeat": "2023-01-14T12:54:54.970596Z",
      "current_task": null
    },
    {
      "pulp_href": "/pulp/api/v3/workers/11fee979-a102-4000-a479-e4df2e187ca0/",
      "pulp_created": "2023-01-14T12:54:44.967084Z",
      "name": "422@1bc295b1c711",
      "last_heartbeat": "2023-01-14T12:54:55.013619Z",
      "current_task": null
    }
  ],
  "online_content_apps": [
    {
      "name": "432@1bc295b1c711",
      "last_heartbeat": "2023-01-14T12:54:59.225545Z"
    },
    {
      "name": "433@1bc295b1c711",
      "last_heartbeat": "2023-01-14T12:54:59.224744Z"
    }
  ],
  "database_connection": {
    "connected": true
  },
  "redis_connection": {
    "connected": false
  },
  "storage": null,
  "content_settings": {
    "content_origin": "http://test-pulp-fedora.c.openshift-gce-devel.internal:8080",
    "content_path_prefix": "/pulp/content/"
  }
}

So, my opinion is that maybe sometimes it takes more time to initialize the entire pulp, hence the errors. Could you give it more time please ?

My suggestions :

  1. Ensure the container are already up and running.
  2. Firstly try to check it with curl localhost:8080/pulp/api/v3/status/ and then use the pulpcore-manager.

Tas-sos avatar Jan 14 '23 19:01 Tas-sos

So, my opinion is that maybe sometimes it takes more time to initialize the entire pulp, hence the errors. Could you give it more time please ?

Ok, thanks for the info - if the tracebacks are expected, I think I am fine (just a small thing that the logs are very ugly and cause conviction that something is broken).

praiskup avatar Jan 16 '23 07:01 praiskup

For the sake of completeness, podman logs pulp shows:

Calling /etc/init/postgres-prepare
2023-01-10 09:29:25.952 UTC [12591] LOG:  redirecting log output to logging collector process
2023-01-10 09:29:25.952 UTC [12591] HINT:  Future log output will appear in directory "log".
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 363, in execute
    settings.INSTALLED_APPS
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py", line 299, in <module>
    from dynaconf import DjangoDynaconf, Validator  # noqa
  File "/usr/local/lib/python3.8/site-packages/dynaconf/__init__.py", line 3, in <module>
    from dynaconf.base import LazySettings  # noqa
  File "/usr/local/lib/python3.8/site-packages/dynaconf/base.py", line 17, in <module>
    from dynaconf.loaders import env_loader
  File "/usr/local/lib/python3.8/site-packages/dynaconf/loaders/env_loader.py", line 8, in <module>
    from dynaconf.vendor.dotenv import cli as dotenv_cli
  File "/usr/local/lib/python3.8/site-packages/dynaconf/vendor/dotenv/cli.py", line 16, in <module>
    @click.option('-f','--file',default=os.path.join(os.getcwd(),'.env'),type=click.Path(exists=_A),help='Location of the .env file, defaults to .env file in current working directory.')
FileNotFoundError: [Errno 2] No such file or directory

This error is related to a dynaconf bug which already have a fix and is expected to be in the next dynaconf release (3.1.12). The same error is reported pulp/pulp-oci-images#404 and pulp/pulpcore#3310

decko avatar Feb 13 '23 17:02 decko

@praiskup can you check if this is happening with the latest version of the pulp image? I'm not seeing them anymore.

decko avatar Jun 09 '23 18:06 decko