packit-service `403 Forbidden` from API endpoints

For a while dashboard has been having issues with rejected API requests because of missing Access-Control-Allow-Origin header.

error

I have to refresh multiple times (it varies, but usually more than 10 times) until an API request succeeds and dashboard becomes usable - once a requests succeeds, the problem goes away until I close the browser tab.

requests

It happens across different OSes with different browser profiles, I use chromium-based browser but I'm quite sure it happened in Firefox as well, though I'm not able to reproduce it with Firefox now.

Sep 16 '24 08:09 nforro

@nforro this has to do with the API missing the header when returning 403. It's nothing to do with the dashboard. Why it's returning 403 I couldn't tell you. Maybe it's a specific pod that is unavailable and returning 403 Forbidden?

Sep 16 '24 10:09 Venefilyn

Makes sense, so there are two issues, one similar to https://github.com/packit/packit-service/issues/1883 and the other needs further investigation.

Sep 16 '24 12:09 nforro

Here is a request sent from the browser (failed ones and successful ones are exactly the same):

fetch("https://stg.packit.dev/api/copr-builds?page=1", {
  "headers": {
    "accept": "*/*",
    "accept-language": "cs-CZ,cs;q=0.9,en;q=0.8",
    "priority": "u=1, i",
    "sec-ch-ua": "\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\"",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "\"Linux\"",
    "sec-fetch-dest": "empty",
    "sec-fetch-mode": "cors",
    "sec-fetch-site": "same-site"
  },
  "referrer": "https://dashboard.stg.packit.dev/",
  "referrerPolicy": "strict-origin-when-cross-origin",
  "body": null,
  "method": "GET",
  "mode": "cors",
  "credentials": "omit"
});

When the request is successful, it's visible in service logs:

172.20.122.9 - - [02/Oct/2024:19:33:50 +0000] "GET /api/copr-builds?page=1 HTTP/2.0" 206 8640

When the request ends with 403 Forbidden, there is nothing in service logs, however there is this in dashboard logs:

[Wed Oct 02 19:31:33.898739 2024] [authz_core:error] [pid 51:tid 62] [remote 172.20.122.9:33218] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8000:1007890000/htdocs/api, referer: https://dashboard.stg.packit.dev/
172.20.122.9 - - [02/Oct/2024:19:31:33 +0000] "GET /api/copr-builds?page=1 HTTP/2.0" 403 199

Oct 03 '24 07:10 nforro

Usually HEADER requests doesn't show in logs IIRC. Can you enable verbose logging for the staging instance to investigate more?

This sounds like a configuration issue with the mod_wsgi settings

client denied by server configuration: /tmp/mod_wsgi-localhost:8000:1007890000/htdocs/api

Possibly related? https://stackoverflow.com/questions/16362462/flask-mod-wsgi-client-denied-by-server-configuration

Oct 03 '24 11:10 Venefilyn

But there should be no requests to the dashboard itself, everything uses Service API

Oct 03 '24 11:10 mfocko

Usually HEADER requests doesn't show in logs IIRC. Can you enable verbose logging for the staging instance to investigate more?

I don't think the browser sends any HEAD requests, but more verbose logging on stage can't hurt, I guess --log-level info should do the trick.

Oct 03 '24 11:10 nforro

Ah huh.. it shouldn't make any requests at all to the dashboard, particularly because I've setup a hard-code for all React API requests to use the import.meta.env.VITE_API_URL, which is set to https://prod.packit.dev/api for prod and it defaults to staging if not set

So it shouldn't direct it to the dashboard. Maybe there is an routing ingress issue in OCP that is not explicitly set?

Oct 03 '24 14:10 Venefilyn

it shouldn't make any requests at all to the dashboard, particularly because I've setup a hard-code for all React API requests to use the import.meta.env.VITE_API_URL, which is set to https://prod.packit.dev/api for prod and it defaults to staging if not set

Yes, the requests are made to stg.packit.dev, even the :authority: pseudo-header is set to stg.packit.dev. Of course it happens on prod too.

Oct 03 '24 14:10 nforro

I don't see anything wrong, so reaching out to the cluster admins.

Reference: https://redhat-internal.slack.com/archives/C04LZLPE7QU/p1730903564686669

Nov 06 '24 11:11 mfocko

Ticket: RITM1933277

Nov 07 '24 13:11 mfocko

Still no reply on the ticket, but I “somehow” managed to get 403 even in browser on the plain endpoint, can't reproduce in shell via curl, also wasn't connected to the VPN, so couldn't check the logs right away (MP+ cluster is available on the VPN, so connecting changes DNS and routing…)

Jan 12 '25 21:01 mfocko

I've noticed that it usually works (at least for the first time) in a browser when I go directly to a "jobs" URL like https://dashboard.packit.dev/jobs/koji or https://dashboard.packit.dev/pipeline, when I try to get there via the menu it's pretty much guaranteed to fail with 403.

Jan 13 '25 08:01 nforro

I've noticed that it usually works (at least for the first time) in a browser when I go directly to a "jobs" URL like https://dashboard.packit.dev/jobs/koji or https://dashboard.packit.dev/pipeline, when I try to get there via the menu it's pretty much guaranteed to fail with 403.

That supports the suggestion from IT ticket that it might be the frontend itself, or based on the fact I got it in the browser, but cannot reproduce via curl, it could be the browser /o\

Jan 13 '25 16:01 mfocko

That supports the suggestion from IT ticket that it might be the frontend itself, or based on the fact I got it in the browser, but cannot reproduce via curl, it could be the browser /o\

Yes, I would suspect the browser as well, but the requests in Developer Tools are correct, and I just verified that with Wireshark:

the HTTP/2 request goes to the correct IP (3.220.240.243 which corresponds to router-external-router-shard.external-router-shard.mpp-e1-prod.9e4s.p1.openshiftapps.com which is a CNAME of prod.packit.dev)
it has :authority: prod.packit.dev, :method: GET and :path: /api/copr-builds?page=1

and yet the response is:

Headers:

:status: 403 Forbidden
content-length: 199
content-type: text/html; charset=iso-8859-1
date: Mon, 13 Jan 2025 17:13:23 GMT
server: Apache

Data:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>

(because the request ends up in the dashboard pod: 172.20.226.11 - - [13/Jan/2025:17:13:23 +0000] "GET /api/copr-builds?page=1 HTTP/2.0" 403 199)

A successfull request performed after a moment looks exactly the same.

Jan 13 '25 17:01 nforro

Do you think logging the entire denied request (including HTTP/IP headers) in dashboard would help in any way?

Jan 14 '25 09:01 nforro

I guess we could try, I wonder if we can log just the 403 to avoid “spam” there.

Jan 14 '25 10:01 mfocko

Update (based on the comments in the ticket and @nforro investigation)

There seems to be a relation to the persistent connections of the HTTP/2 (initial connection with TLS handshake is done to the dashboard; SNI is dashboard.packit.dev and with the same SNI it makes a request to the Packit Service API, thus resulting in the request being routed to the dashboard rather than the production API itself, since the routing for TLS Passthrough connections is done based on the SNI).

Related to packit/dashboard#425 Related to packit/packit-service#2450

TODO

[x] Return 421 instead of 403 on the dashboard' side, or
[ ] Revert aforementioned PRs, or
[ ] From the front end query the dashboard and have the dashboard API query the Packit Service.

`403 Forbidden` from API endpoints

Update (based on the comments in the ticket and @nforro investigation)

TODO

Related links