flux bootstrapping not working for bitbucket server with none-root baseurl / not handling http 301 response

[cdm-arm]

Describe the bug

flux version 0.30.2 running against a Bitbucket server fails to get project listing. Reason seems to be that neither a base url can be provided to flux nor is the 301 redirect honoured.

✗ failed to create new Git repository "fluxcd": failed to get Git organization: failed to get organization "KUBDEP": get project failed, unable to unmarshal repository list json: invalid character '<' looking for beginning of value

Looking into the http request with mitmproxy reveals that flux makes the assumption that the rest api can be reached under /rest/api

Steps to reproduce

1. Install a Bitbucket server and configure the base url like below

2. Set the token: export BITBUCKET_TOKEN=xxxxxxxxxxx

3. Run the flux cli to bootstrap flux flux bootstrap bitbucket-server \
   --hostname=bitbucket.example.com
   --username=fluxcd
   --token-auth
   --owner=KUBDEP
   --repository=fluxcd
   --branch=master

4. Receive the error ► connecting to bitbucket.example.com ✗ failed to create new Git repository "fluxcd": failed to get Git organization: failed to get organization "KUBDEP": get project failed, unable to unmarshal repository list json: invalid character '<' looking for beginning of value

Expected behavior

bootstrapping works. Either through the 301 redirect being honoured or a parameter with baseurl for flux.

Screenshots and recordings

1. Response received if complete url is used

➜ ~ curl -v https://bitbucket.example.com/bitbucket/rest/api/1.0/projects?name=exampleproject

• Trying 10.xx.xx.xx:443...
• Connected to bitbucket.example.com (10.xx.xx.xx) port 443 (#0) ....
• SSL certificate verify ok.

GET /bitbucket/rest/api/1.0/projects?name=exampleproject HTTP/1.1 Host: bitbucket.example.com User-Agent: curl/7.79.1 Accept: /

• Mark bundle as not supporting multiuse < HTTP/1.1 200 < Server: nginx/1.xx < Date: Mon, 23 May 2022 14:28:46 GMT < Content-Type: application/json;charset=UTF-8 < Transfer-Encoding: chunked < Connection: keep-alive < X-AREQUESTID: xxxxxxxxx < X-ASEN: SEN-xxxxxxxx < Cache-Control: no-cache, no-transform < Vary: accept-encoding,x-auserid,cookie,x-ausername,accept-encoding < X-Content-Type-Options: nosniff <
• Connection #0 to host git.cdm.smis.ch left intact {"size":0,"limit":25,"isLastPage":true,"values":[],"start":0}

2. Response that flux receives (via proxy)

OS / Distro

MacOS 12.1

Flux version

0.30.2

Flux check

➜ ~ flux check
► checking prerequisites ✔ Kubernetes 1.21.6+vmware.2 >=1.20.6-0 ► checking controllers ✔ all checks passed

Git provider

Bitbucket

Container Registry provider

N/A

Additional context

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[cdm-arm]

Workaround is possible if you have ssh access to the bitbucket server itself and you can create own dns entries.

1. Create a copy of the bitbucket nginx config
2. Listen on an alternate server_name (here gitapi.xxx.ch instead of bitbucket.xxx.ch).
3. Remove the root (/) location
4. Amend the /bitbucket location to be root /
5. Change the proxy_pass to include whatever is configured in your Bitbucket server (here bitbucket/ )
6. Enable the site in your nginx and restart

server {
        listen 443 ssl;
        server_name gitapi.xxx.ch;

        include /etc/nginx/conf.d/xxx_ch.ssl.conf;
        access_log /var/log/nginx/api.access.log;
        error_log /var/log/nginx/api.error.log;

        location / {
                proxy_pass      http://localhost:7990/bitbucket/;
                proxy_set_header        X-Forwarded-Host $host;
                proxy_set_header        X-Forwarded-Server $host;
                proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header    X-Real-IP $remote_addr;
                proxy_redirect          off;
                client_max_body_size 0;
                proxy_buffering off;
        }
}

[souleb]

Hi, thanks for filling this issue. The client should definitely follow to up to 10 redirects for 301, 302, 303. Let me look at the code.

[souleb]

Hi @aek-arm in the current code, we compute a URI for the request. In our case it would be /rest/api/1.0/projects/?name=project_name. When the append the it tho the base URL (set here as --hostname=bitbucket.example.com) to get the URL that we would use for the request.

Then we use an httpClient that follows redirect with the default setup.

[cdm-arm]

Hi @souleb - Sorry for late reply, I finally looked into it with your explanation. There are 2 issues from what I can see:

1. When computing the URI flux makes the assumption that the rest api is available at root level. This is not mandatory for an on premise bitbucket server. Anyhow at least Bitbucket does a redirect. Guess having a baseURL would be smoother. Will open a ticket for that.

2. The httpClient seems really to follow the redirect that is returned, as can be seen in the above screenshot of the mitmproxy. A 301 permanent redirect is sent, but with the location set incorrectly to Location: https://bitbucket.example.com/bitbucket instead of a correct Location: https://bitbucket.example.com/bitbucket/rest/api/1.0/projects?name=exampleproject This causes the httpclient to receive a html page with code 200, that then fails to be parsed as ✗ failed to create new Git repository "flux": failed to get Git organization: failed to get organization "exampleproject": get project failed, unable to unmarshal repository list json: invalid character '<' looking for beginning of value

Maybe an improvement would be to display in that case the response?

Anyway I'm closing this issue as it is caused by a misconfiguration of Bitbucket server.

Note for future readers that encountered the same behaviour with out of the box Bitbucket:

A proper NGINX config block should include the request_uri like this:

        location / {
                return 301 https://bitbucket.example.com/bitbucket$request_uri;
        }