flux2
flux2 copied to clipboard
fluxcd
GitRepository support for fallback url
Describe the bug
On my current company we are using flux on all of our clusters. We started to move all of our services from ec2 instances to k8s. The latest thing that we moved was our self hosted Gitlab.
The problem was when we moved gitlab to k8s, flux deleted everything that it manages. We had to disable flux on our infrastructure cluster and deploy everything manually until we address this problem. I had the oportunity to talk about it with Scott Rigby during the Europe Kubecon 2022 and he explained to me why it happened and that flux expect to have git source somewhere else. Since Flux needs Gitlab to deploy Gitlab, is a chicken egg problem.
I have a proposal to addres this problem, since we need flux to deploy and use our Gitlab I have been thinking on having a fall back url.
The following is a very quick (and probably could be improved proposal):
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
name: podinfo
namespace: default
spec:
interval: 5m0s
url: https://github.com/stefanprodan/podinfo
fallbackUrl: https://gitlab.com/stefanprodan/podinfo
ref:
branch: master
The idea is to have flux using the normal url as it's currently doing. But if for some reason the main url fails, it will try with the fallbackUrl which should be a mirror of the main url. Also this parameter could be optional so it will not break the current usage.
I'm also willing to try to implement this but wanted to know what do you think first.
Steps to reproduce
N/A
Expected behavior
N/A
Screenshots and recordings
No response
OS / Distro
N/A
Flux version
N/A
Flux check
N/A
Git provider
No response
Container Registry provider
No response
Additional context
No response
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
@stefanprodan @hiddeco Hello there, Think that this would be a good solution for the use case. This will allow Gitlab to create mirrors outside, for example in GitHub. What is your opinion about it?
I'm not in favour if this, you can't have a backup by just providing a different URL, for SSH the deploy keys are unique per repo so the auth will fail, same with HTTPS access, a GitHub token will not work on GitLab.
Yes, you are right, I didn't thought about the different ssh keys and https tokens. I have a second proposal that could fit here. Instead of having a fallback url defined on the gitrepository, what about having two different gitrepositories, each one configured to his own git server. But allowing to have the fallback logic at kustomization level, let me add an example of how it could be:
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
name: podinfo-github
namespace: default
spec:
interval: 5m0s
url: https://github.com/stefanprodan/podinfo
ref:
branch: master
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
name: podinfo-gitlab
namespace: default
spec:
interval: 5m0s
url: https://gitlab.com/stefanprodan/podinfo
ref:
branch: master
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: podinfo
namespace: flux-system
spec:
interval: 10m0s
path: ./
prune: true
sourceRef:
kind: GitRepository
name: podinfo-gitlab
fallbackSourceRef:
kind: GitRepository
name: podinfo-github
The fallbackSourceRef is the first name that came to my mind but could be changed, I'm open to ideas about this :smiley:
You can do the swap in Git, include the first one in the kustomization.yaml, then before you migrate, replace it with the backup. Another option is to use a kustomize patch, you sync both secrets, then before the migration you enable a patch that changes the URL and the secret name.
Hello @stefanprodan The use case I detect here is not intended for migrations, but for the case you use a Git, for example GitLab onprem and this Gitlab mirrors some needed repos to another git, for example GitHub or Gitea. If Kustomization or HelmRelease admits to have a fallback GitRepository, you could reconcile from one and, if this is not up, from the other. What do you think about it?
Hi! Sorry for the long time without answer, I missed the notification. I agree with @achetronic about the use case. We discovered it during the migration, but the idea is to found a way to have repository redundancy in the case where one is down for any reason.
What do you think about it @stefanprodan ?
How would the controller confirm that the fallback is actually a mirror of the primary URL? As I am missing any details around this in the proposal, and think that as soon as you start looking into this, the solution is no longer viable due to the complexity of the required logic.
To be honest, I have not thought about those deep details but, would it be possible to calculate a kind of hash of the code stored on one repository and another to check? Anyway, may be you are right and the implementation details make it not to be worth. What do you think @hiddeco??
The guys are right. We have experienced today GitLab outage for 3 hours, and if there were some fallback option it would save us from potential problems in case of GitLab unavailability.
https://twitter.com/gitlabstatus/status/1566067672315736064?cxt=HHwWgMDR6YSQ5bsrAAAA
The guys are right. We have experienced today GitLab outage for 3 hours, and if there were some fallback option it would save us from potential problems in case of GitLab unavailability.
https://twitter.com/gitlabstatus/status/1566067672315736064?cxt=HHwWgMDR6YSQ5bsrAAAA
Well, the point is, imagine you have one repo on GitHub and another on GitLab, and they have the same content. The problem is about being completely sure about it, because if they differs, you will break all the deployments related. So do you have any suggestion about how to easily handle this problem?
Remember the CR called GitRepository is managed by the Source Controller, and if I am not wrong, it is in charge of doing some validations to confirm and expose the content to the other controllers
I am of opinion that if outages of your Git provider are an issue to you, there are sufficient solutions out there to mitigate this issue. E.g. by mirroring your repository to a high-available solution you are in control over yourself, setting up a Persistent Volume to back the source-controller so it continues to maintain the last observed state (if it gets evicted from a node), etc.
Given this, I continue to not see any need to add fallback behavior to the controller, as there are sufficient solutions out there which can help you deal with this issue without us introducing complexity and API changes to the controller.
I am of opinion that if outages of your Git provider are an issue to you, there are sufficient solutions out there to mitigate this issue. E.g. by mirroring your repository to a high-available solution you are in control over yourself, setting up a Persistent Volume to back the source-controller so it continues to maintain the last observed state (if it gets evicted from a node), etc.
Given this, I continue to not see any need to add fallback behavior to the controller, as there are sufficient solutions out there which can help you deal with this issue without us introducing complexity and API changes to the controller.
Thank you for the clarification on this topic and the example for it :)