Utils.remoteFileExist broken for http(s) on instances with login requirements

[kk-chesse]

Follow up from #1698.
Utils.remoteFileExist uses axios to check if a file exists on http(s) repositories. Now with #1699 it always fails, due to being redirected to the signin page.

Additional context Gitlab 18.6 configured to require login on http(s)

I'm not sure if this is a use case you want to support. If not, at least the behaviour should be documented.

I looked into potential fixes involving using the git client to avoid login problem.

1. Use git archive with https, this requires atleast git 2.44 (released February 2024). Unfortunately this didn't work for me error: RPC failed; HTTP 422 curl 22 The requested URL returned error: 422
2. Do a sparse checkout git 2.25 (released January 2020). While a lot more complicated, it could unify the logic for all protocols supported by git

[kk-chesse]

Looking into the usage of Utils.remoteFileExist I see it's only used once in ParserIncludes.init to determine the directory structure of the component repository.

Thinking out loud, would it be feasible to refactor the ParserIncludes.init to do a sparse checkout with both templates/<component>.yml and templates/<component>/template.yml and check later which file to use? This would reduce the required calls to the server to just 1, instead of 2 or 3.

[ANGkeith]

can you show the output of your redacted git remote -v ?

i think you're currently using https for your repo right ?

could you update it to ssh instead ?

ie.

# Don't use https
git remote -v
origin  https://gitlab.com/example/apple.git (fetch)
origin  https://gitlab.com/example/apple.git (push)

# use this instead
git remote -v
[email protected]:ANGkeith/shhh.git
origin  [email protected]:example/apple.git (fetch)
origin  [email protected]:example/apple.git (push)

Then your issue should be fixed.

---

yeah, i totally forgotten. The remoteFileExist will not work for repos that are private

[kk-chesse]

git remote -v
origin  https://<domain>/group/project.git (fetch)
origin  https://<domain>/group/project.git (push)

I can't use ssh. Any operations just sits idle for a few minutes and then timeouts. I think because our coporate firewall blocks it.

I've have a proof of concept to use sparse checkouts for components #1701

[peterbbeu]

I can't use ssh. Any operations just sits idle for a few minutes and then timeouts. I think because our coporate firewall blocks it.

Have you tried using SSH over port 443?

https://about.gitlab.com/blog/gitlab-dot-com-now-supports-an-alternate-git-plus-ssh-port/

Or are you not on gitlab.com?

[kk-chesse]

It's a selfhosted coporate Gitlab instance, accessible only via https.