rancher
Deploy the same repo via kustomize to different namespaces on the same cluster
My repository is structured as follows. However, when I provide a path to the dev folder. Fleet does not seem to find the base folder? I'd like to use this repo to deploy multiple instances of my app to different namespaces on the same cluster. Currently we are using ArgoCD to do this by pointing it to the overlay folder we want to deploy. For example, the dev folder. When we use Argo, it finds the base folder. In Fleet, however, I can't seem to find a way to do the same thing. I do see a way to use the same repo to deploy to multiple clusters using different overlay folders by using a fleet.yaml file. However, it does not look like the fleet.yaml file will support the same cluster with different namespaces. Am I missing something in the documentation?
Error Produced
error while running post render on files: accumulating resources: accumulateFile "accumulating resources from '../../base': '../../base' doesn't exist", loader.New "error loading ../../base with git: url lacks host: ../../base, dir: '../../base' doesn't exist, get: invalid source string: ../../base"
Repo Structure:
- base
- overlays
- dev
- kustomization.yaml
- stage
- prod
- temp
- dev
gz#14201
Yes, I've tried. It doesn't work. It looks like Fleet requires the base directory to be a child of the path. If this is true. Then kustomize in fleet won't work for multiple instances in the same cluster. Very disappointing as this is basic 101 type stuff. If I can apply my kustomization.yaml file using the following. Fleet should be able to do the same.
kubectl apply -k overlays\dev -n dev
kustomization.yaml located in overlays\dev
bases:
- ../../base
resources:
- ingress.yaml
patches:
- image-tag.yaml
- replica-count.yaml
configMapGenerator:
- name: myapp
behavior: replace
envs:
- configs/settings.env
It seems like the default Fleet repo scanning algorithm described here: https://github.com/rancher/fleet/blob/master/docs/gitrepo-structure.md#how-repos-are-scanned should be more configurable via the GitRepo resource to allow explicit specification of one or more fleet.yaml file paths. This would allow the Fleet GitRepo resource to behave more like the ArgoCD Application resource to support this type of scenario. This would allow you to create one Fleet GitRepo resource per overlay that explicitly references a single fleet.yaml.
I just used the UI to build the manifest. But here is what it generated.
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
creationTimestamp: "2021-01-05T20:47:40Z"
generation: 3
name: caas-demo
namespace: apps7
resourceVersion: "301964103"
selfLink: /apis/fleet.cattle.io/v1alpha1/namespaces/apps7/gitrepos/caas-demo
uid: 2b23e586-992a-40a8-ac01-5711c60d15d7
spec:
branch: fleet
clientSecretName: my-secret
forceSyncGeneration: 2
paths:
- /caas/overlays/demo
repo: repourl
targetNamespace: demo
targets:
- clusterSelector:
matchLabels:
management.cattle.io/cluster-name: c-vzcs4
status:
commit: 6b44fec07e1e40dfd53947bda53a48b5e88f7613
conditions:
- lastUpdateTime: "2021-01-05T20:50:18Z"
message: 'ErrApplied(1) [Bundle caas-demo-caas-overlays-demo: error while running
post render on files: accumulating resources: accumulateFile "accumulating resources
from ''../../base'': ''../../base'' doesn''t exist", loader.New "error loading
../../base with git: url lacks host: ../../base, dir: ''../../base'' doesn''t
exist, get: invalid source string: ../../base"]'
status: "False"
type: Ready
- lastUpdateTime: "2021-01-05T20:50:18Z"
status: "True"
type: Accepted
- lastUpdateTime: "2021-01-05T20:49:51Z"
status: "False"
type: Reconciling
- lastUpdateTime: "2021-01-05T20:47:40Z"
status: "False"
type: Stalled
- lastUpdateTime: "2021-01-05T20:49:51Z"
status: "True"
type: Synced
desiredReadyClusters: 1
display:
error: true
message: 'error while running post render on files: accumulating resources: accumulateFile
"accumulating resources from ''../../base'': ''../../base'' doesn''t exist",
loader.New "error loading ../../base with git: url lacks host: ../../base, dir:
''../../base'' doesn''t exist, get: invalid source string: ../../base"'
readyBundleDeployments: 0/1
state: ErrApplied
gitJobStatus: Current
observedGeneration: 3
readyClusters: 0
resourceCounts:
desiredReady: 0
missing: 0
modified: 0
notReady: 0
orphaned: 0
ready: 0
unknown: 0
waitApplied: 0
resourceErrors:
- 'error while running post render on files: accumulating resources: accumulateFile
"accumulating resources from ''../../base'': ''../../base'' doesn''t exist", loader.New
"error loading ../../base with git: url lacks host: ../../base, dir: ''../../base''
doesn''t exist, get: invalid source string: ../../base"'
summary:
desiredReady: 1
errApplied: 1
nonReadyResources:
- bundleState: ErrApplied
message: 'error while running post render on files: accumulating resources:
accumulateFile "accumulating resources from ''../../base'': ''../../base''
doesn''t exist", loader.New "error loading ../../base with git: url lacks
host: ../../base, dir: ''../../base'' doesn''t exist, get: invalid source
string: ../../base"'
name: caas-demo-caas-overlays-demo
ready: 0
type: fleet.cattle.io.gitrepo
@stbraley Does the fleet example in the following git repo adequately capture your use case: https://github.com/ron1/fleet-examples/tree/fleet-mc-kustomize-multi-namespace-example/multi-cluster/kustomize-multi-namespace?
@StrongMonkey Is the above example a development-centric use case Fleet would aspire to support? Today, we must use ArgoCD deployed via Fleet to support these development-centric, multi-namespace workloads.
@ron1 I think so with one caveat. It needs to ignore folders in the overlay directories that don't have a path specified in the GitRepo resource. We have some deployments in the git repository that are not intended to be deployed to clusters.
@stbraley @StrongMonkey I made a minor change to the GitRepo in file: https://github.com/ron1/fleet-examples/blob/fleet-mc-kustomize-multi-namespace-example/multi-cluster/kustomize-multi-namespace/README.md to allow multiple fleetFiles per path. A GitRepo path with one or more explicit fleetFiles that include helm and/or kustomize directives should not search for unspecified raw yaml manifests. So, I would expect Fleet to ignore "...deployments in the git repository that are not intended to be deployed to clusters".
Just found a solution to this problem. may be try doing the following in your fleet.yaml to get same deployment working in multiple nameaspaces or same namespaces. this worked for me on fleet 0..6. hope this can be helpful for you guys as well
namespace: lab
targetCustomizations:
- name: dev
helm:
values:
replicas: 1
clusterSelector:
matchLabels:
env: dev
namespace: lab2
targetCustomizations:
- name: dev
helm:
values:
replicas: 3
clusterSelector:
matchLabels:
env: dev```
We have multiple kustomize overlays in a single git repository that have many shared bases. We want to deploy each overlay from a unique GitRepo, into a specific namespace, where each GitRepo would point to a specific tag. This way we can deploy specific overlays/namespaces, one at a time, by changing the revision that items's tag points to.
The fleetFiles that @ron1 suggested would be a solution.
Using multiple paths does not work because we cannot share any common bases between the different paths.
Without some solution we cannot use fleet.
Is there any progress on this issue? @ron1 implementation looks good to me. Is there anything I can contribute to make this move forward?
For @ron1's example we would need a PR that changes the bundlereader, which is part of the CLI.
The current code takes the GitRepo.spec.paths and passes them to the fleet CLI to create bundles, like this: fleet apply --targets-file=/run/config/targets.yaml -- name path1 path2. That can be run separately.
Then CreateBundles scans each path recursively, detecting and creating "bundles" as documented in How Repos are Scanned.
I would prefer an addition to the current implementation that does not scan recursively. The detection makes Fleet easy to use, but sometimes hard to debug. Ideally I could define the bundles in the GitRepo explicitly:
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
name: example
spec:
repo: ...
bundles:
- path1
- path2
However, to solve this issue, we also need to be able to include another directory at the same level or below the bundle directory.
The mentioned approach with fleetFiles to specify the fleet.yaml would work. We could even put the fleet.yamls in subfolders?
.
├── base
├── overlays
│ ├── dev
│ └── prod
├── fleet1.yaml # points to overlays/dev
├── fleet2.yaml # points to overlays/prod
I wonder about the semantics, this would translate to:
- take
path:as a base dir for the bundle's resources - generate a bundle for every
fleetFilesentry
spec:
repo: https://github.com/rancher/fleet-examples
paths:
- path: multi-cluster/kustomize-multi-namespace
fleetFiles:
- fleet-dev-dev1.yaml
- fleet-prod.yaml
I would prefer, to not collide with the scanner for paths: and flat over nested:
spec:
repo: ...
bundles:
- path: multi-cluster/kustomize-multi-namespace
options: fleet-dev-dev1.yaml
- path: multi-cluster/kustomize-multi-namespace
options: fleet-prod.yaml
fleet.yaml is eventually turned into BundleSpec and ImageScans.
@manno For us, we would prefer to store the fleet file in a sub-folder but use the root of the repo as the base for path so things can be more organized. I'm OK with flat over nested.
.
├── base
├── overlays
├── dev
| ├── fleet.yaml # points to this overlay
└── prod
├── fleet.yaml # points to this overlay
Or
├── fleetFiles
| ├── fleet-dev-dev1.yaml # points to overlays/dev
| ├── fleet-prod.yaml # points to overlays/prod
├── base
├── overlays
├── dev
└── prod
Then deploy a specific fleetfile like:
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: dev1
namespace: fleet-default
spec:
repo: https://github.com/rancher/fleet-examples
paths:
- path: multi-cluster/kustomize-multi-namespace
fleetFile: fleetFiles/fleet-dev-dev1.yaml
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: prod
namespace: fleet-default
spec:
repo: https://github.com/rancher/fleet-examples
paths:
- path: multi-cluster/kustomize-multi-namespace
fleetFile: fleetFiles/fleet-prod.yaml
```
Additional QA
Problem
Fleet is having some problems when deploying kustomize resources. Specially when the user wants to deploy certain kustomizations inside a folder and not others.
Solution
Fleet introduces a new way of defining bundles that is user-driven and that does not actively scan the folder(s) passed.
It just adds all the resources found in the given base folder and a user-defined fleet.yaml could also be used.
You can find the new bundle definition in the docs (look for the bundles field)
Testing
Engineering Testing
Manual Testing
Used the fleet-test-data repository, deploying the driven folder using the new way of defining bundles this way:
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: driven
namespace: fleet-local
spec:
repo: https://github.com/0xavi0/fleet-test-data
bundles:
- base: driven/helm
- base: driven/simple
- base: driven/kustomize
options: dev.yaml
- base: driven/kustomize
options: test.yaml
Verify that Fleet creates the 4 expected bundles.
Verify that Fleet does not create the production Bundle referenced by the production.yaml found in the kustomize folder.
Verify that the kustomizations are correct (for example, check that suffix for each ConfigMap is the one defined in each kustomize folder)
QA Testing Considerations
I would try to generate Bundles in different ways and verify that they are being created are expected. You can also play with kustomize setups similar to the proposed one and verify that they are applied successfully.
System Information
| Rancher Version | Fleet Version |
|---|---|
| v2.12-356748b8653d770f9b79a0bd15e28415f4d9ffa3-head | 107.0.0+up0.13.0-alpha.3 |
Test Scenario's
| Sr. No. | Test Cases | Test Result |
|---|---|---|
| 1 | Test GitRepo creates bundles specified under bundles: without option: field |
✅ |
| 2 | Test GitRepo creates bundles specified under bundles: with option: field |
✅ |
| 3 | Test update GitRepo by removing prod.yaml from option and verify that prod bundle should not be created. |
✅ |
| 4 | Test update GitRepo by adding test.yaml under option and verify that prod bundle should not be created. |
✅ |
Steps followed
Scenario 1
- Created a GitRepo with Fleet-test-data using path driven
- Wait for a resources to be created.
- Verified that all bundles were created without any error.
[YAML] 1. GitRepo with bundle option in spec
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
name: test-bundle
namespace: fleet-local
spec:
branch: master
bundles:
- base: driven/helm
- base: driven/simple
- base: driven/kustomize
paths:
- driven
repo: https://github.com/rancher/fleet-test-data
[Screenshot] 4 Bundles created with suffix helm, simple, kustomize
Scenario 2
- Created a GitRepo with Fleet-test-data using path driven
- Wait for a resources to be created.
- Verified that all bundles were created except test bundle without any error.
[YAML] 2. GitRepo with bundle option in spec
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
name: test-bundle-options
namespace: fleet-local
spec:
branch: master
bundles:
- base: driven/helm
- base: driven/simple
- base: driven/kustomize
options: dev.yaml
- base: driven/kustomize
options: prod.yaml
paths:
- driven
repo: https://github.com/rancher/fleet-test-data
[Screenshot] 5 Bundles created with suffix helm, simple, kustomize[prod, dev]
Scenario 3
- Created a GitRepo with Fleet-test-data using path driven
- Wait for a resources to be created.
- Verified that all bundles were created except test bundle without any error.
- Remove
prod.yamlline from theoption:under- base: driven/kustomizeby editing GitRepo. - Verified that bundle with prod suffix were removed and all bundles were intact.
[YAML] 3. GitRepo with bundle option in spec
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
name: test-bundle-options
namespace: fleet-local
spec:
branch: master
bundles:
- base: driven/helm
- base: driven/simple
- base: driven/kustomize
options: dev.yaml
- base: driven/kustomize
paths:
- driven
repo: https://github.com/rancher/fleet-test-data
[Screenshot] 5 Bundles created with suffix helm, simple, kustomize[dev]
Scenario 4
- Created a GitRepo with Fleet-test-data using path driven
- Wait for a resources to be created.
- Verified that all bundles were created except test bundle without any error.
- Add
test.yamlline from theoption:under- base: driven/kustomizeby editing GitRepo. - Verified that bundle with test suffix were added along with other bundles.
[YAML] 4. GitRepo with bundle option in spec
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
name: test-bundle-options
namespace: fleet-local
spec:
branch: master
bundles:
- base: driven/helm
- base: driven/simple
- base: driven/kustomize
options: dev.yaml
- base: driven/kustomize
options: test.yaml
paths:
- driven
repo: https://github.com/rancher/fleet-test-data
