waldur-mastermind icon indicating copy to clipboard operation
waldur-mastermind copied to clipboard
verified publisher icon waldur

[Openstack] Not possible to synchronise a new tenant offer when using a domain other than the one used by Openstack administrator

Open Ketchup31 opened this issue 11 months ago • 17 comments

Hi,

I am trying to setup a new offering for a openstack tenant.

I created:

  1. A new domain
  2. A new project in this domain
  3. A new user with Admin rights
  4. Affect the user to this domain

I can login to Openstack with credentials I created in steps above.

I set in Waldur values requested in attributes but I get this error message in "old" admin portal:

Forbidden('You are not authorized to perform the requested action: identity:list_domains. (HTTP 403) (Request-ID: req-a0c252f8-6203-46fd-8654-9e603d0aa215)')

If I set the values with openstack administrator account, it works. But i do not want to use the domain of the admin account, and wish to create a dedicated domain where all projects created via Waldur will be hosted.

Do you have any idea about what i did wrong or what I can do?

Thank you for your help.

Best regards, Pascal

Waldur latest (04/04/2025 evening) docker-compose install

waldur-mastermind-worker  | [2025-04-05 06:27:42,323: ERROR/ForkPoolWorker-8] Task waldur_core.core.tasks.IndependentBackendMethodTask[784d6e2e-fe12-4d44-80a2-a591138a1aeb] raised unexpected: OpenStackBackendError(Forbidden('You are not authorized to perform the requested action: identity:list_domains. (HTTP 403) (Request-ID: req-9d5e9d2b-7d62-469d-9542-6594a911de22)'))
waldur-mastermind-worker  | Traceback (most recent call last):
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 248, in pull_tenants
waldur-mastermind-worker  |     backend_tenants = keystone.projects.list(domain=self._get_domain())
waldur-mastermind-worker  |                                                     ^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 281, in _get_domain
waldur-mastermind-worker  |     return keystone.domains.find(name=self.settings.domain or "Default")
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 75, in func
waldur-mastermind-worker  |     return f(*args, **new_kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 438, in find
waldur-mastermind-worker  |     elements = self._list(
waldur-mastermind-worker  |                ^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 125, in _list
waldur-mastermind-worker  |     resp, body = self.client.get(url, **kwargs)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 673, in get
waldur-mastermind-worker  |     return self.request(url, 'GET', **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 657, in request
waldur-mastermind-worker  |     resp = self._request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 294, in _request
waldur-mastermind-worker  |     return self.session.request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/session.py", line 1118, in request
waldur-mastermind-worker  |     raise exceptions.from_response(resp, method, url)
waldur-mastermind-worker  | keystoneauth1.exceptions.http.Forbidden: You are not authorized to perform the requested action: identity:list_domains. (HTTP 403) (Request-ID: req-9d5e9d2b-7d62-469d-9542-6594a911de22)
waldur-mastermind-worker  |
waldur-mastermind-worker  | During handling of the above exception, another exception occurred:
waldur-mastermind-worker  |
waldur-mastermind-worker  | Traceback (most recent call last):
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
waldur-mastermind-worker  |     R = retval = fun(*args, **kwargs)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
waldur-mastermind-worker  |     return self.run(*args, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/core/tasks.py", line 111, in run
waldur-mastermind-worker  |     result = self.execute(instance, *self.args, **self.kwargs)
waldur-mastermind-worker  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/core/tasks.py", line 270, in execute
waldur-mastermind-worker  |     getattr(backend, backend_method)(*args, **kwargs)
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/structure/backend.py", line 53, in sync
waldur-mastermind-worker  |     self.pull_resources()
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 242, in pull_resources
waldur-mastermind-worker  |     self.pull_tenants()
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 250, in pull_tenants
waldur-mastermind-worker  |     raise OpenStackBackendError(e)
waldur-mastermind-worker  | waldur_openstack.exceptions.OpenStackBackendError: You are not authorized to perform the requested action: identity:list_domains. (HTTP 403) (Request-ID: req-9d5e9d2b-7d62-469d-9542-6594a911de22)
waldur-mastermind-worker  | [2025-04-05 06:27:42,325: INFO/MainProcess] Task waldur_core.core.tasks.ErrorStateTransitionTask[b30f467a-6771-405a-9d5f-259615b73c09] received
waldur-mastermind-worker  | [2025-04-05 06:27:42,355: ERROR/ForkPoolWorker-1] Instance: structure.servicesettings:1.
waldur-mastermind-worker  | Error: Forbidden('You are not authorized to perform the requested action: identity:list_domains. (HTTP 403) (Request-ID: req-9d5e9d2b-7d62-469d-9542-6594a911de22)').
waldur-mastermind-worker  | Traceback (most recent call last):
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 248, in pull_tenants
waldur-mastermind-worker  |     backend_tenants = keystone.projects.list(domain=self._get_domain())
waldur-mastermind-worker  |                                                     ^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 281, in _get_domain
waldur-mastermind-worker  |     return keystone.domains.find(name=self.settings.domain or "Default")
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 75, in func
waldur-mastermind-worker  |     return f(*args, **new_kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 438, in find
waldur-mastermind-worker  |     elements = self._list(
waldur-mastermind-worker  |                ^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 125, in _list
waldur-mastermind-worker  |     resp, body = self.client.get(url, **kwargs)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 673, in get
waldur-mastermind-worker  |     return self.request(url, 'GET', **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 657, in request
waldur-mastermind-worker  |     resp = self._request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 294, in _request
waldur-mastermind-worker  |     return self.session.request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/session.py", line 1118, in request
waldur-mastermind-worker  |     raise exceptions.from_response(resp, method, url)
waldur-mastermind-worker  | keystoneauth1.exceptions.http.Forbidden: You are not authorized to perform the requested action: identity:list_domains. (HTTP 403) (Request-ID: req-9d5e9d2b-7d62-469d-9542-6594a911de22)
waldur-mastermind-worker  |
waldur-mastermind-worker  | During handling of the above exception, another exception occurred:
waldur-mastermind-worker  |
waldur-mastermind-worker  | Traceback (most recent call last):
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
waldur-mastermind-worker  |     R = retval = fun(*args, **kwargs)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
waldur-mastermind-worker  |     return self.run(*args, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/core/tasks.py", line 111, in run
waldur-mastermind-worker  |     result = self.execute(instance, *self.args, **self.kwargs)
waldur-mastermind-worker  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/core/tasks.py", line 270, in execute
waldur-mastermind-worker  |     getattr(backend, backend_method)(*args, **kwargs)
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/structure/backend.py", line 53, in sync
waldur-mastermind-worker  |     self.pull_resources()
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 242, in pull_resources
waldur-mastermind-worker  |     self.pull_tenants()
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 250, in pull_tenants
waldur-mastermind-worker  |     raise OpenStackBackendError(e)
waldur-mastermind-worker  | waldur_openstack.exceptions.OpenStackBackendError: You are not authorized to perform the requested action: identity:list_domains. (HTTP 403) (Request-ID: req-9d5e9d2b-7d62-469d-9542-6594a911de22)
waldur-mastermind-worker  | NoneType: None
waldur-mastermind-worker  | [2025-04-05 06:27:42,360: INFO/ForkPoolWorker-1] State of ServiceSettings instance `XXX Tenant (OpenStack)` (PK: 1) changed from UPDATING to ERRED, with method `set_erred`
waldur-mastermind-worker  | [2025-04-05 06:27:42,367: INFO/ForkPoolWorker-1] Task waldur_core.core.tasks.ErrorStateTransitionTask[b30f467a-6771-405a-9d5f-259615b73c09] succeeded in 0.041079137998167425s: None

Ketchup31 avatar Apr 04 '25 19:04 Ketchup31

hm, one option is to relax the validation logic and to trust user provided domain name. Currently it tries to fetch it - and that provides a meaningful error message if domain is e.g. incorrectly spelled, but indeed, leads to a need for additional permission.

But otherwise, creation of projects and local users in that domain with that admin user works?

livenson avatar Apr 05 '25 08:04 livenson

Hello,

Thank you for your support.

If I use the credentials created for this domain, I can create a project and users. I can also login with user credentials to use the project created in that domain. So at your question "But otherwise, creation of projects and local users in that domain with that admin user works?" I can say Yes

Ketchup31 avatar Apr 05 '25 09:04 Ketchup31

ok, then I guess we can just add a handler to use domain as string in case permission error is raised. not a very big change, will be added soon.

livenson avatar Apr 05 '25 09:04 livenson

Thank you so much. Have a great week-end !

Ketchup31 avatar Apr 05 '25 09:04 Ketchup31

Ok, pushed some fix - https://github.com/waldur/waldur-mastermind/commit/674a8e0cf93eb9a2c8da73da1fc647bf11003d6c - would be great if you could validate on your deployment. Images should be ready in ~30mins.

livenson avatar Apr 05 '25 10:04 livenson

OK. Your are amazing !!! Thank you so much. I try it in 30 minutes as I have an appointement now. I will keep you inform of the results. If it is OK I could continue my integration of Waldur in our system this week-end !

Ketchup31 avatar Apr 05 '25 10:04 Ketchup31

OK test done. It seems that the issue is still there.

Here what I did:

  1. Delete waldur images from docker
  2. launch #sudo docker compose up -d
  3. Go to offerings->Edit->Synchronize
  4. Status = Erred

For your info, I keep the domain name. Do I need to change anything in my values?

Here the logs from mastermind-worker:

waldur-mastermind-worker  | [2025-04-05 11:28:06,353: INFO/MainProcess] Task waldur_core.core.tasks.IndependentBackendMethodTask[f4f23155-6510-4031-8eb3-72fa73023413] received
waldur-mastermind-worker  | [2025-04-05 11:28:08,386: INFO/ForkPoolWorker-8] State of ServiceSettings instance `XXX Tenant (OpenStack)` (PK: 1) changed from UPDATE_SCHEDULED to UPDATING, with method `begin_updating`
waldur-mastermind-worker  | [2025-04-05 11:28:08,951: WARNING/ForkPoolWorker-8] User is not authorized to list domains. Using domain name as string: gdccustomers_domain
waldur-mastermind-worker  | [2025-04-05 11:28:09,032: ERROR/ForkPoolWorker-8] Task waldur_core.core.tasks.IndependentBackendMethodTask[f4f23155-6510-4031-8eb3-72fa73023413] raised unexpected: OpenStackBackendError(Forbidden('You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-45bf4cc7-4961-43c2-a319-95ac60b16d39)'))
waldur-mastermind-worker  | Traceback (most recent call last):
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 248, in pull_tenants
waldur-mastermind-worker  |     backend_tenants = keystone.projects.list(domain=self._get_domain())
waldur-mastermind-worker  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/v3/projects.py", line 134, in list
waldur-mastermind-worker  |     projects = super(ProjectManager, self).list(
waldur-mastermind-worker  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 75, in func
waldur-mastermind-worker  |     return f(*args, **new_kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 395, in list
waldur-mastermind-worker  |     return self._list(
waldur-mastermind-worker  |            ^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 125, in _list
waldur-mastermind-worker  |     resp, body = self.client.get(url, **kwargs)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 673, in get
waldur-mastermind-worker  |     return self.request(url, 'GET', **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 657, in request
waldur-mastermind-worker  |     resp = self._request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 294, in _request
waldur-mastermind-worker  |     return self.session.request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/session.py", line 1118, in request
waldur-mastermind-worker  |     raise exceptions.from_response(resp, method, url)
waldur-mastermind-worker  | keystoneauth1.exceptions.http.Forbidden: You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-45bf4cc7-4961-43c2-a319-95ac60b16d39)
waldur-mastermind-worker  |
waldur-mastermind-worker  | During handling of the above exception, another exception occurred:
waldur-mastermind-worker  |
waldur-mastermind-worker  | Traceback (most recent call last):
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
waldur-mastermind-worker  |     R = retval = fun(*args, **kwargs)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
waldur-mastermind-worker  |     return self.run(*args, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/core/tasks.py", line 111, in run
waldur-mastermind-worker  |     result = self.execute(instance, *self.args, **self.kwargs)
waldur-mastermind-worker  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/core/tasks.py", line 270, in execute
waldur-mastermind-worker  |     getattr(backend, backend_method)(*args, **kwargs)
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/structure/backend.py", line 53, in sync
waldur-mastermind-worker  |     self.pull_resources()
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 242, in pull_resources
waldur-mastermind-worker  |     self.pull_tenants()
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 250, in pull_tenants
waldur-mastermind-worker  |     raise OpenStackBackendError(e)
waldur-mastermind-worker  | waldur_openstack.exceptions.OpenStackBackendError: You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-45bf4cc7-4961-43c2-a319-95ac60b16d39)
waldur-mastermind-worker  | [2025-04-05 11:28:09,044: INFO/MainProcess] Task waldur_core.core.tasks.ErrorStateTransitionTask[57a9ac54-d747-4b96-abe1-63945d5395e8] received
waldur-mastermind-worker  | [2025-04-05 11:28:09,072: ERROR/ForkPoolWorker-8] Instance: structure.servicesettings:1.
waldur-mastermind-worker  | Error: Forbidden('You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-45bf4cc7-4961-43c2-a319-95ac60b16d39)').
waldur-mastermind-worker  | Traceback (most recent call last):
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 248, in pull_tenants
waldur-mastermind-worker  |     backend_tenants = keystone.projects.list(domain=self._get_domain())
waldur-mastermind-worker  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/v3/projects.py", line 134, in list
waldur-mastermind-worker  |     projects = super(ProjectManager, self).list(
waldur-mastermind-worker  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 75, in func
waldur-mastermind-worker  |     return f(*args, **new_kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 395, in list
waldur-mastermind-worker  |     return self._list(
waldur-mastermind-worker  |            ^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 125, in _list
waldur-mastermind-worker  |     resp, body = self.client.get(url, **kwargs)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 673, in get
waldur-mastermind-worker  |     return self.request(url, 'GET', **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 657, in request
waldur-mastermind-worker  |     resp = self._request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 294, in _request
waldur-mastermind-worker  |     return self.session.request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/session.py", line 1118, in request
waldur-mastermind-worker  |     raise exceptions.from_response(resp, method, url)
waldur-mastermind-worker  | keystoneauth1.exceptions.http.Forbidden: You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-45bf4cc7-4961-43c2-a319-95ac60b16d39)
waldur-mastermind-worker  |
waldur-mastermind-worker  | During handling of the above exception, another exception occurred:
waldur-mastermind-worker  |
waldur-mastermind-worker  | Traceback (most recent call last):
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
waldur-mastermind-worker  |     R = retval = fun(*args, **kwargs)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
waldur-mastermind-worker  |     return self.run(*args, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/core/tasks.py", line 111, in run
waldur-mastermind-worker  |     result = self.execute(instance, *self.args, **self.kwargs)
waldur-mastermind-worker  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/core/tasks.py", line 270, in execute
waldur-mastermind-worker  |     getattr(backend, backend_method)(*args, **kwargs)
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/structure/backend.py", line 53, in sync
waldur-mastermind-worker  |     self.pull_resources()
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 242, in pull_resources
waldur-mastermind-worker  |     self.pull_tenants()
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 250, in pull_tenants
waldur-mastermind-worker  |     raise OpenStackBackendError(e)
waldur-mastermind-worker  | waldur_openstack.exceptions.OpenStackBackendError: You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-45bf4cc7-4961-43c2-a319-95ac60b16d39)
waldur-mastermind-worker  | NoneType: None
waldur-mastermind-worker  | [2025-04-05 11:28:09,079: INFO/ForkPoolWorker-8] State of ServiceSettings instance `XXX Tenant (OpenStack)` (PK: 1) changed from UPDATING to ERRED, with method `set_erred`
waldur-mastermind-worker  | [2025-04-05 11:28:09,088: INFO/ForkPoolWorker-8] Task waldur_core.core.tasks.ErrorStateTransitionTask[57a9ac54-d747-4b96-abe1-63945d5395e8] succeeded in 0.04296560998773202s: None

Ketchup31 avatar Apr 05 '25 11:04 Ketchup31

can you please check what's the value of the domain that you have? identity:list_projects is for sure a needed permission. I would assume that domain admin account would have it.

livenson avatar Apr 05 '25 11:04 livenson

I've also added a workaround - https://github.com/waldur/waldur-mastermind/commit/f1bd71530601f9ab19a309d467d9268af9124b8d - but not sure it will work for all operations.

anyway, 30mins and you can check.

livenson avatar Apr 05 '25 11:04 livenson

OK. Thank you. I am re-checking all my settings in Openstack too.

Ketchup31 avatar Apr 05 '25 11:04 Ketchup31

OK I see the new docker. I launch my tests. Thank you.

Ketchup31 avatar Apr 05 '25 12:04 Ketchup31

The issue is still there.

What is strange is I can loggin in horizon with the same credentials defined in openstack and waldur. For your information, i declared this user in a project of the domain and I get the error hereunder.

If I do not define any project by default for this user. I get the error. (Logs at bottom of this message)

Here the logs (With default project for the user):

[2025-04-05 12:38:22,021: INFO/MainProcess] Task waldur_core.core.tasks.IndependentBackendMethodTask[382f86d6-df4e-447f-afeb-42cdc04af949] received
waldur-mastermind-worker  | [2025-04-05 12:38:24,054: INFO/ForkPoolWorker-8] State of ServiceSettings instance `XXX Tenant (OpenStack)` (PK: 1) changed from UPDATE_SCHEDULED to UPDATING, with method `begin_updating`
waldur-mastermind-worker  | [2025-04-05 12:38:24,434: INFO/ForkPoolWorker-8] Starting to pull tenants for service settings XXX Tenant (OpenStack)
waldur-mastermind-worker  | [2025-04-05 12:38:24,514: WARNING/ForkPoolWorker-8] User is not authorized to list domains. Using domain name as string: gdccustomers_domain. Error: You are not authorized to perform the requested action: identity:list_domains. (HTTP 403) (Request-ID: req-6ed998e8-a304-4206-9b72-26523fd42548)
waldur-mastermind-worker  | [2025-04-05 12:38:24,590: WARNING/ForkPoolWorker-8] User is not authorized to list all projects. This might be expected if the user only has access to specific projects. Error: You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-794ed4eb-f45c-4fa8-a3b8-2ddbecc55f51)
waldur-mastermind-worker  | [2025-04-05 12:38:24,637: ERROR/ForkPoolWorker-8] Failed to list accessible projects: You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-fd9f4719-8d9b-481d-9f49-8c9c1e25fe58)
waldur-mastermind-worker  | [2025-04-05 12:38:24,665: INFO/MainProcess] Task waldur_core.core.tasks.ErrorStateTransitionTask[8d41b845-2730-4652-b602-9315d099bdaf] received
waldur-mastermind-worker  | [2025-04-05 12:38:24,664: ERROR/ForkPoolWorker-8] Task waldur_core.core.tasks.IndependentBackendMethodTask[382f86d6-df4e-447f-afeb-42cdc04af949] raised unexpected: OpenStackBackendError(Forbidden('You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-fd9f4719-8d9b-481d-9f49-8c9c1e25fe58)'))
waldur-mastermind-worker  | Traceback (most recent call last):
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 251, in pull_tenants
waldur-mastermind-worker  |     backend_tenants = keystone.projects.list(domain=domain)
waldur-mastermind-worker  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/v3/projects.py", line 134, in list
waldur-mastermind-worker  |     projects = super(ProjectManager, self).list(
waldur-mastermind-worker  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 75, in func
waldur-mastermind-worker  |     return f(*args, **new_kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 395, in list
waldur-mastermind-worker  |     return self._list(
waldur-mastermind-worker  |            ^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 125, in _list
waldur-mastermind-worker  |     resp, body = self.client.get(url, **kwargs)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 673, in get
waldur-mastermind-worker  |     return self.request(url, 'GET', **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 657, in request
waldur-mastermind-worker  |     resp = self._request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 294, in _request
waldur-mastermind-worker  |     return self.session.request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/session.py", line 1118, in request
waldur-mastermind-worker  |     raise exceptions.from_response(resp, method, url)
waldur-mastermind-worker  | keystoneauth1.exceptions.http.Forbidden: You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-794ed4eb-f45c-4fa8-a3b8-2ddbecc55f51)
waldur-mastermind-worker  |
waldur-mastermind-worker  | During handling of the above exception, another exception occurred:
waldur-mastermind-worker  |
waldur-mastermind-worker  | Traceback (most recent call last):
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 260, in pull_tenants
waldur-mastermind-worker  |     backend_tenants = keystone.projects.list()
waldur-mastermind-worker  |                       ^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/v3/projects.py", line 134, in list
waldur-mastermind-worker  |     projects = super(ProjectManager, self).list(
waldur-mastermind-worker  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 75, in func
waldur-mastermind-worker  |     return f(*args, **new_kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 395, in list
waldur-mastermind-worker  |     return self._list(
waldur-mastermind-worker  |            ^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 125, in _list
waldur-mastermind-worker  |     resp, body = self.client.get(url, **kwargs)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 673, in get
waldur-mastermind-worker  |     return self.request(url, 'GET', **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 657, in request
waldur-mastermind-worker  |     resp = self._request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 294, in _request
waldur-mastermind-worker  |     return self.session.request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/session.py", line 1118, in request
waldur-mastermind-worker  |     raise exceptions.from_response(resp, method, url)
waldur-mastermind-worker  | keystoneauth1.exceptions.http.Forbidden: You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-fd9f4719-8d9b-481d-9f49-8c9c1e25fe58)
waldur-mastermind-worker  |
waldur-mastermind-worker  | During handling of the above exception, another exception occurred:
waldur-mastermind-worker  |
waldur-mastermind-worker  | Traceback (most recent call last):
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
waldur-mastermind-worker  |     R = retval = fun(*args, **kwargs)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
waldur-mastermind-worker  |     return self.run(*args, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/core/tasks.py", line 111, in run
waldur-mastermind-worker  |     result = self.execute(instance, *self.args, **self.kwargs)
waldur-mastermind-worker  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/core/tasks.py", line 270, in execute
waldur-mastermind-worker  |     getattr(backend, backend_method)(*args, **kwargs)
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/structure/backend.py", line 53, in sync
waldur-mastermind-worker  |     self.pull_resources()
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 242, in pull_resources
waldur-mastermind-worker  |     self.pull_tenants()
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 267, in pull_tenants
waldur-mastermind-worker  |     raise OpenStackBackendError(e2)
waldur-mastermind-worker  | waldur_openstack.exceptions.OpenStackBackendError: You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-fd9f4719-8d9b-481d-9f49-8c9c1e25fe58)
waldur-mastermind-worker  | [2025-04-05 12:38:24,700: ERROR/ForkPoolWorker-1] Instance: structure.servicesettings:1.
waldur-mastermind-worker  | Error: Forbidden('You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-fd9f4719-8d9b-481d-9f49-8c9c1e25fe58)').
waldur-mastermind-worker  | Traceback (most recent call last):
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 251, in pull_tenants
waldur-mastermind-worker  |     backend_tenants = keystone.projects.list(domain=domain)
waldur-mastermind-worker  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/v3/projects.py", line 134, in list
waldur-mastermind-worker  |     projects = super(ProjectManager, self).list(
waldur-mastermind-worker  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 75, in func
waldur-mastermind-worker  |     return f(*args, **new_kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 395, in list
waldur-mastermind-worker  |     return self._list(
waldur-mastermind-worker  |            ^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 125, in _list
waldur-mastermind-worker  |     resp, body = self.client.get(url, **kwargs)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 673, in get
waldur-mastermind-worker  |     return self.request(url, 'GET', **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 657, in request
waldur-mastermind-worker  |     resp = self._request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 294, in _request
waldur-mastermind-worker  |     return self.session.request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/session.py", line 1118, in request
waldur-mastermind-worker  |     raise exceptions.from_response(resp, method, url)
waldur-mastermind-worker  | keystoneauth1.exceptions.http.Forbidden: You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-794ed4eb-f45c-4fa8-a3b8-2ddbecc55f51)
waldur-mastermind-worker  |
waldur-mastermind-worker  | During handling of the above exception, another exception occurred:
waldur-mastermind-worker  |
waldur-mastermind-worker  | Traceback (most recent call last):
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 260, in pull_tenants
waldur-mastermind-worker  |     backend_tenants = keystone.projects.list()
waldur-mastermind-worker  |                       ^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/v3/projects.py", line 134, in list
waldur-mastermind-worker  |     projects = super(ProjectManager, self).list(
waldur-mastermind-worker  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 75, in func
waldur-mastermind-worker  |     return f(*args, **new_kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 395, in list
waldur-mastermind-worker  |     return self._list(
waldur-mastermind-worker  |            ^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneclient/base.py", line 125, in _list
waldur-mastermind-worker  |     resp, body = self.client.get(url, **kwargs)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 673, in get
waldur-mastermind-worker  |     return self.request(url, 'GET', **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 657, in request
waldur-mastermind-worker  |     resp = self._request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 294, in _request
waldur-mastermind-worker  |     return self.session.request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/session.py", line 1118, in request
waldur-mastermind-worker  |     raise exceptions.from_response(resp, method, url)
waldur-mastermind-worker  | keystoneauth1.exceptions.http.Forbidden: You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-fd9f4719-8d9b-481d-9f49-8c9c1e25fe58)
waldur-mastermind-worker  |
waldur-mastermind-worker  | During handling of the above exception, another exception occurred:
waldur-mastermind-worker  |
waldur-mastermind-worker  | Traceback (most recent call last):
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
waldur-mastermind-worker  |     R = retval = fun(*args, **kwargs)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
waldur-mastermind-worker  |     return self.run(*args, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/core/tasks.py", line 111, in run
waldur-mastermind-worker  |     result = self.execute(instance, *self.args, **self.kwargs)
waldur-mastermind-worker  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/core/tasks.py", line 270, in execute
waldur-mastermind-worker  |     getattr(backend, backend_method)(*args, **kwargs)
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/structure/backend.py", line 53, in sync
waldur-mastermind-worker  |     self.pull_resources()
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 242, in pull_resources
waldur-mastermind-worker  |     self.pull_tenants()
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 267, in pull_tenants
waldur-mastermind-worker  |     raise OpenStackBackendError(e2)
waldur-mastermind-worker  | waldur_openstack.exceptions.OpenStackBackendError: You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-fd9f4719-8d9b-481d-9f49-8c9c1e25fe58)
waldur-mastermind-worker  | NoneType: None
waldur-mastermind-worker  | [2025-04-05 12:38:24,707: INFO/ForkPoolWorker-1] State of ServiceSettings instance `XXX Tenant (OpenStack)` (PK: 1) changed from UPDATING to ERRED, with method `set_erred`
waldur-mastermind-worker  | [2025-04-05 12:38:24,716: INFO/ForkPoolWorker-1] Task waldur_core.core.tasks.ErrorStateTransitionTask[8d41b845-2730-4652-b602-9315d099bdaf] succeeded in 0.04970839500310831s: None

Here the logs (Without default project for the user):

waldur-mastermind-worker  | [2025-04-05 12:47:23,684: INFO/MainProcess] Task waldur_core.core.tasks.IndependentBackendMethodTask[a4fb203d-c01a-4321-9c0d-a25d9b0d0acc] received
waldur-mastermind-worker  | [2025-04-05 12:47:25,717: INFO/ForkPoolWorker-8] State of ServiceSettings instance `XXX Tenant (OpenStack)` (PK: 1) changed from UPDATE_SCHEDULED to UPDATING, with method `begin_updating`
waldur-mastermind-worker  | [2025-04-05 12:47:25,909: ERROR/ForkPoolWorker-8] Task waldur_core.core.tasks.IndependentBackendMethodTask[a4fb203d-c01a-4321-9c0d-a25d9b0d0acc] raised unexpected: NotFound('Could not find token: gAAAAABn8SQFqBV2XBqFCEPTTfbyQB0QEKeX8RHT27f96BqUFJm_Auptp24BnRwk-rteuZZO0MdN6fLfYduLEV-knsFHWudzLVPg5jRU2ov4H2JInVZisN_jqaKT7ADeQf2KhgqKC7VSy3SOxK4F7d58kmaGIf5TCQI2_O6HnzEHEBvoGDdoljs. (HTTP 404) (Request-ID: req-7b31d78c-044d-4e3b-9dd4-0d684a168759)')
waldur-mastermind-worker  | Traceback (most recent call last):
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
waldur-mastermind-worker  |     R = retval = fun(*args, **kwargs)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
waldur-mastermind-worker  |     return self.run(*args, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/core/tasks.py", line 111, in run
waldur-mastermind-worker  |     result = self.execute(instance, *self.args, **self.kwargs)
waldur-mastermind-worker  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/core/tasks.py", line 270, in execute
waldur-mastermind-worker  |     getattr(backend, backend_method)(*args, **kwargs)
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/structure/backend.py", line 52, in sync
waldur-mastermind-worker  |     self.pull_service_properties()
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 238, in pull_service_properties
waldur-mastermind-worker  |     self.pull_service_settings_quotas()
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 2847, in pull_service_settings_quotas
waldur-mastermind-worker  |     stats = nova.hypervisor_stats.statistics()
waldur-mastermind-worker  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/novaclient/v2/hypervisors.py", line 140, in statistics
waldur-mastermind-worker  |     return self._get("/os-hypervisors/statistics", "hypervisor_statistics")
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/novaclient/base.py", line 356, in _get
waldur-mastermind-worker  |     resp, body = self.api.client.get(url)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 673, in get
waldur-mastermind-worker  |     return self.request(url, 'GET', **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/novaclient/client.py", line 74, in request
waldur-mastermind-worker  |     resp, body = super(SessionClient, self).request(url,
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 657, in request
waldur-mastermind-worker  |     resp = self._request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 294, in _request
waldur-mastermind-worker  |     return self.session.request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/session.py", line 1108, in request
waldur-mastermind-worker  |     auth_headers = self.get_auth_headers(auth)
waldur-mastermind-worker  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/session.py", line 1387, in get_auth_headers
waldur-mastermind-worker  |     return auth.get_headers(self)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/plugin.py", line 124, in get_headers
waldur-mastermind-worker  |     token = self.get_token(session)
waldur-mastermind-worker  |             ^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/identity/base.py", line 91, in get_token
waldur-mastermind-worker  |     return self.get_access(session).auth_token
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/identity/base.py", line 139, in get_access
waldur-mastermind-worker  |     self.auth_ref = self.get_auth_ref(session)
waldur-mastermind-worker  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/identity/v3/base.py", line 240, in get_auth_ref
waldur-mastermind-worker  |     resp = session.post(
waldur-mastermind-worker  |            ^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/session.py", line 1334, in post
waldur-mastermind-worker  |     return self.request(url, 'POST', **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/session.py", line 1118, in request
waldur-mastermind-worker  |     raise exceptions.from_response(resp, method, url)
waldur-mastermind-worker  | keystoneauth1.exceptions.http.NotFound: Could not find token: gAAAAABn8SQFqBV2XBqFCEPTTfbyQB0QEKeX8RHT27f96BqUFJm_Auptp24BnRwk-rteuZZO0MdN6fLfYduLEV-knsFHWudzLVPg5jRU2ov4H2JInVZisN_jqaKT7ADeQf2KhgqKC7VSy3SOxK4F7d58kmaGIf5TCQI2_O6HnzEHEBvoGDdoljs. (HTTP 404) (Request-ID: req-7b31d78c-044d-4e3b-9dd4-0d684a168759)
waldur-mastermind-worker  | [2025-04-05 12:47:25,912: INFO/MainProcess] Task waldur_core.core.tasks.ErrorStateTransitionTask[220dec10-4091-4a67-a304-b30214e5bc5f] received
waldur-mastermind-worker  | [2025-04-05 12:47:25,945: ERROR/ForkPoolWorker-1] Instance: structure.servicesettings:1.
waldur-mastermind-worker  | Error: Could not find token: gAAAAABn8SQFqBV2XBqFCEPTTfbyQB0QEKeX8RHT27f96BqUFJm_Auptp24BnRwk-rteuZZO0MdN6fLfYduLEV-knsFHWudzLVPg5jRU2ov4H2JInVZisN_jqaKT7ADeQf2KhgqKC7VSy3SOxK4F7d58kmaGIf5TCQI2_O6HnzEHEBvoGDdoljs. (HTTP 404) (Request-ID: req-7b31d78c-044d-4e3b-9dd4-0d684a168759) (HTTP 404).
waldur-mastermind-worker  | Traceback (most recent call last):
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
waldur-mastermind-worker  |     R = retval = fun(*args, **kwargs)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
waldur-mastermind-worker  |     return self.run(*args, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/core/tasks.py", line 111, in run
waldur-mastermind-worker  |     result = self.execute(instance, *self.args, **self.kwargs)
waldur-mastermind-worker  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/core/tasks.py", line 270, in execute
waldur-mastermind-worker  |     getattr(backend, backend_method)(*args, **kwargs)
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_core/structure/backend.py", line 52, in sync
waldur-mastermind-worker  |     self.pull_service_properties()
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 238, in pull_service_properties
waldur-mastermind-worker  |     self.pull_service_settings_quotas()
waldur-mastermind-worker  |   File "/usr/src/waldur/src/waldur_openstack/backend.py", line 2847, in pull_service_settings_quotas
waldur-mastermind-worker  |     stats = nova.hypervisor_stats.statistics()
waldur-mastermind-worker  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/novaclient/v2/hypervisors.py", line 140, in statistics
waldur-mastermind-worker  |     return self._get("/os-hypervisors/statistics", "hypervisor_statistics")
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/novaclient/base.py", line 356, in _get
waldur-mastermind-worker  |     resp, body = self.api.client.get(url)
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 673, in get
waldur-mastermind-worker  |     return self.request(url, 'GET', **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/novaclient/client.py", line 74, in request
waldur-mastermind-worker  |     resp, body = super(SessionClient, self).request(url,
waldur-mastermind-worker  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 657, in request
waldur-mastermind-worker  |     resp = self._request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/adapter.py", line 294, in _request
waldur-mastermind-worker  |     return self.session.request(url, method, **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/session.py", line 1108, in request
waldur-mastermind-worker  |     auth_headers = self.get_auth_headers(auth)
waldur-mastermind-worker  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/session.py", line 1387, in get_auth_headers
waldur-mastermind-worker  |     return auth.get_headers(self)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/plugin.py", line 124, in get_headers
waldur-mastermind-worker  |     token = self.get_token(session)
waldur-mastermind-worker  |             ^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/identity/base.py", line 91, in get_token
waldur-mastermind-worker  |     return self.get_access(session).auth_token
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/identity/base.py", line 139, in get_access
waldur-mastermind-worker  |     self.auth_ref = self.get_auth_ref(session)
waldur-mastermind-worker  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/identity/v3/base.py", line 240, in get_auth_ref
waldur-mastermind-worker  |     resp = session.post(
waldur-mastermind-worker  |            ^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/session.py", line 1334, in post
waldur-mastermind-worker  |     return self.request(url, 'POST', **kwargs)
waldur-mastermind-worker  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
waldur-mastermind-worker  |   File "/usr/local/lib/python3.11/site-packages/keystoneauth1/session.py", line 1118, in request
waldur-mastermind-worker  |     raise exceptions.from_response(resp, method, url)
waldur-mastermind-worker  | keystoneauth1.exceptions.http.NotFound: Could not find token: gAAAAABn8SQFqBV2XBqFCEPTTfbyQB0QEKeX8RHT27f96BqUFJm_Auptp24BnRwk-rteuZZO0MdN6fLfYduLEV-knsFHWudzLVPg5jRU2ov4H2JInVZisN_jqaKT7ADeQf2KhgqKC7VSy3SOxK4F7d58kmaGIf5TCQI2_O6HnzEHEBvoGDdoljs. (HTTP 404) (Request-ID: req-7b31d78c-044d-4e3b-9dd4-0d684a168759)
waldur-mastermind-worker  | NoneType: None
waldur-mastermind-worker  | [2025-04-05 12:47:25,955: INFO/ForkPoolWorker-1] State of ServiceSettings instance `XXX Tenant (OpenStack)` (PK: 1) changed from UPDATING to ERRED, with method `set_erred`
waldur-mastermind-worker  | [2025-04-05 12:47:25,965: INFO/ForkPoolWorker-1] Task waldur_core.core.tasks.ErrorStateTransitionTask[220dec10-4091-4a67-a304-b30214e5bc5f] succeeded in 0.04905937900184654s: None

Ketchup31 avatar Apr 05 '25 12:04 Ketchup31

ok, I think this is a bit more complicated task and needs proper reproduction for a fix.

Can you share openstack distro and exact actions that were made in order to create a separate domain and admin user in that domain?

livenson avatar Apr 06 '25 07:04 livenson

Hi,

Openstack distro is USSURY.

What I did:

  1. With admin user I was logged in admin_domain domain.
  2. -> go in Domain tab and created a new domain and select "set domain context" for this domain
  3. -> go in User tab and create a new user
  4. -> go in Project and created a new project with user just created above with Admin rights
  5. Logged out
  6. Logged in to the new Project/Domain/User account created above
  7. I created a new project manually and it worked.

Ketchup31 avatar Apr 06 '25 09:04 Ketchup31

Hi,

I saw with our openstack expert. If we want that user is able to list domains, we need to give a scope at System level, which is not really what we want to do. I do not know why Waldur needs to list the domains (perhaps to check if the domain exists?), But maybe another operation/process could be implemented?

To be able to list domain we had to update our .rc file like hereunder: .../... export OS_SYSTEM_SCOPE=all unset OS_PROJECT_ID unset OS_PROJECT_NAME unset OS_PROJECT_DOMAIN_NAME .../...

Is our Openstack version (WALLABY and not USSURY as mentioned above) is too old?

Thank you for your support.

Best regards,

Ketchup31 avatar Apr 07 '25 16:04 Ketchup31

Hi,

both versions should be ok. I suspect there is some function call that basically triggers wrong API call. Sorry, this investigation is a bit larger work. Do we have any support contract? I could prioritise the fix then. Otherwise the task is in the backlog and will be solved at normal pace.

livenson avatar Apr 07 '25 17:04 livenson

Hi, Thank you for your answer. No unfortunately we do not have any support contract. We should have to to wait. Best regards. Thanx

Ketchup31 avatar Apr 07 '25 18:04 Ketchup31