"Unauthorized: new_update__POST failed permission check", even when authenticated

[TomasTomecek]

We have recently run into this problem: https://github.com/martinpitt/python-dbusmock/issues/152#issue-1309471879

When Packit submits a new bodhi update using the .save() method, it now fails with:

bodhi.client.bindings.BodhiClientException: {"status": "error", "errors": [{"location": "body", "name": "HTTPForbidden", "description": "Unauthorized: new_update__POST failed permission check"}]}

This happens after we migrated to the Bodhi 6 OIDC authentication.

It's strange because I can see that everything seems to be in place on our side:

• bodhi_client has a CSRF token. When I set it to None, Bodhi client correctly gets a new one.
• bodhi_client.username says packit, which is the correct username
• bodhi_client.oidc.tokens is populated:

  {'access_token': 'wont-share', 'refresh_token': 'wont-share-either', 'expires_in': 3600, 
  'id_token': 'neither-this', 'token_type': 'Bearer', 'expires_at': 1658246335}
  

• After running .clear_auth(), the client correctly obtains new tokens

After I refreshed everything, the error is still the same.

Am I missing something? Is there something new why Packit cannot create an update for python-dbusmock even though it has commit access in the dist-git repo?

[TomasTomecek]

Even ipsilon says we're authenticated correctly:

>>> bodhi_client.oidc.metadata["userinfo_endpoint"]
'https://id.fedoraproject.org/openidc/UserInfo'

>>> response = bodhi_client.oidc.client.get(b.oidc.metadata["userinfo_endpoint"])
>>> response.json()
{'sub': 'packit', 'groups': ['fedorabugs', 'signed_fpca', 'git-packit-team', 'packager', 'ipausers', 'fedora-contributor'], 'email': '[email protected]', 'locale': 'en-US', 'zoneinfo': 'UTC', 'website': 'https://packit.dev', 'preferred_username': 'packit', 'nickname': 'packit', 'name': 'Packit Bot'}

[TomasTomecek]

I just went through this again, step by step (new build, create the update via API) and can confirm am getting still the same error.

I'm wondering if there is some more relevant info in bodhi server log that would tell if there is something wrong with the auth verification.

[nirik]

I assume this is still happening? Can you duplicate on a current build so we can check logs?

[TomasTomecek]

I was away for the past 10 days, sorry for the delay.

Tried right now with my personal FAS account and the result is still the same:

2022-10-03 09:54:37.517 utils.py          ERROR  You are using Bodhi 6 client. There is an issue with creating updates using this version: https://github.com/fedora-infra/bodhi/issues/4660
Traceback (most recent call last):
  File "/home/tt/g/packit/packit/packit/distgit.py", line 498, in create_bodhi_update
    result = bodhi_client.save(**save_kwargs)
  File "/home/tt/g/fedora-infra/bodhi/bodhi-client/bodhi/client/bindings.py", line 128, in wrapper
    result = method(*args, **kwargs)
  File "/home/tt/g/fedora-infra/bodhi/bodhi-client/bodhi/client/bindings.py", line 331, in save
    return self.send_request('updates/', verb='POST', auth=True,
  File "/home/tt/g/fedora-infra/bodhi/bodhi-client/bodhi/client/bindings.py", line 238, in send_request
    raise BodhiClientException(response.text)
bodhi.client.bindings.BodhiClientException: {"status": "error", "errors": [{"location": "body", "name": "HTTPForbidden", "description": "Unauthorized: new_update__POST failed permission check"}]}

UTC time 2022-10-03 07:54:37.517

I got successfully authenticated using OIDC but the .save call failed. I tried to create the update via web interface and it passed fine: https://bodhi.fedoraproject.org/updates/FEDORA-2022-9cbc4f2c65 (I revoked it afterwards).

I'd appreciate any server-side logs that would help us track down what's wrong here.

[nirik]

And of course the bodhi pod restarted a few hours ago...

@abompard any ideas here?

[abompard]

Hey, I'll have a look. @TomasTomecek could you please do it again? As kevin said the logs of last time aren't visible anymore (or maybe Openshift has a log history?)

[TomasTomecek]

Just triggered it again, timestamp '2022-10-05 15:26:32.540 UTC'

[abompard]

Bodhi's logs just have Unauthorized: new_update__POST failed permission check :-/ I'll check Ipsilon to see if there's something weird there

[TomasTomecek]

@abompard is the boss!! He figured out this is actually a problem on packit's side as we change the authentication mechanism to kerberos, but there is a problem with storing auth cookies.

Closing, thank you so much, Aurélien!