kube-linter
kube-linter copied to clipboard
stackrox
[FEATURE_REQUEST] Do not warn about dangling service linked to deployment or statefulset without a pod
Description of the problem/feature request Service definitions with a label selector matching a deployment or statefulset definition without any pod definition are generating a dangling service warning in kube-linter because the dangling service check appears to only concern itself with pods.
Description of the existing behavior vs. expected behavior Consider the following service definition:
apiVersion: v1
kind: Service
metadata:
name: appnameA
labels:
app: appnameALabel
spec:
ports:
- name: http
port: 8080
targetPort: 8080
selector:
app: appnameALabel
type: ClusterIP
And the following (partial) deployment definition:
apiVersion: apps/v1
kind: Deployment
metadata:
name: appnameA
labels:
app: appnameALabel
spec:
replicas: 1
selector:
matchLabels:
app: appnameALabel
When running kube-linter, we get the following:
kubernetes/60-services/appnameA-svc.yaml: (object: <no namespace>/appnameA /v1, Kind=Service) no pods found matching service labels (map[app:appnameALabel]) (check: dangling-service, remediation: Make sure your service's selector correctly matches the labels on one of your deployments.)
Additional context The warning message is technically correct, there is no pod which matches the service label. However this is not a dangling service, and should not generate a warning. We currently annotate for this but we are looking to improve the kubernetes definitions and this is one that sticks out because it's doesn't seem to be a real problem.
Hi @tspearconquest, thanks for filing this issue. I will admit I've never seen a deployment definition without a pod spec. How does that work? If you kubectl create that, will it create pods with some default spec?
Hey Viswajith, I checked in the namespace just now. It looks like it does create a pod with the same spec (or nearly the same, as I could not see any difference) as the deployment.
The definition I provided was only a partial example for demonstration purposes.
In our full setup, we have a CI pipeline which builds the image from our Dockerfile and the app source code, then runs kube-linter, and then deploys the app to a kubernetes cluster. The kubernetes definition is relatively simple but I cannot share the full spec as it is internal to my organization.
The deployment spec defines an image and the auto-generated pod uses that image.
Ohh, got it. I was really confused. 😅
Here's a very minimal example I put together. In this example, kube-linter does not flag the service as a dangling service. You can see it does not find an error.
kube-linter lint - <<EOF --do-not-auto-add-defaults --include dangling-service
apiVersion: apps/v1
kind: Deployment
metadata:
name: test
spec:
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: test
spec:
containers:
- image: nginx:latest
name: nginx
---
apiVersion: v1
kind: Service
metadata:
name: test
labels:
app: test
spec:
ports:
- name: http
port: 8080
targetPort: 8080
selector:
app: test
type: ClusterIP
EOF
Can you help me modify this example so as to reproduce the problem you're seeing?
Hi Viswajith,
When I run kube-linter in the way you have done with a heredoc, using our internal definitions, it is not reporting the issue.
However, when I place them in separate files (deployment.yaml and service.yaml, for example) and run kube-linter lint . then I can see the warning.
Is kube-linter understanding that the deployment.yaml is part of the same context as service.yaml? It should do that. There should be no difference between your heredoc and my 2 files.
Hmm, it should understand that it is part of the same context, and it does. I just tested it that way:
viswa:~/kube-linter-demo/yamls/temp (docker-desktop)$ ls
dep.yaml svc.yaml
viswa:~/kube-linter-demo/yamls/temp (docker-desktop)$ cat dep.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: test
spec:
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: test
spec:
containers:
- image: nginx:latest
name: nginx
viswa:~/kube-linter-demo/yamls/temp (docker-desktop)$ cat svc.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: test
spec:
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: test
spec:
containers:
- image: nginx:latest
name: nginx
---
apiVersion: v1
kind: Service
metadata:
name: test
labels:
app: test
spec:
ports:
- name: http
port: 8080
targetPort: 8080
selector:
app: test
type: ClusterIP
viswa:~/kube-linter-demo/yamls/temp (docker-desktop)$ kube-linter lint . --do-not-auto-add-defaults --include dangling-service
KubeLinter 0.2.1-0-gc53952b8e3
No lint errors found!
Your svc.yaml contains the deployment a second time...
Ah, my bad. I just fixed and retried. Same result though...
viswa:~/kube-linter-demo/yamls/temp (docker-desktop)$ ls
dep.yaml svc.yaml
viswa:~/kube-linter-demo/yamls/temp (docker-desktop)$ cat dep.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: test
spec:
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: test
spec:
containers:
- image: nginx:latest
name: nginx
viswa:~/kube-linter-demo/yamls/temp (docker-desktop)$ cat svc.yaml
apiVersion: v1
kind: Service
metadata:
name: test
labels:
app: test
spec:
ports:
- name: http
port: 8080
targetPort: 8080
selector:
app: test
type: ClusterIP
viswa:~/kube-linter-demo/yamls/temp (docker-desktop)$ kube-linter lint . --do-not-auto-add-defaults --include dangling-service
KubeLinter 0.2.1-0-gc53952b8e3
No lint errors found!
Great, I have reproduced the issue on my end with these 2 manifests. In my setup, we segregate the deployment definition and the service definition into subfolders and run kube-linter against the parent folder.
~/gitlab/workspaces/test ❯ ls
deploy/ svc/
~/gitlab/workspaces/test ❯ ls -R
deploy/ svc/
./deploy:
dep.yaml
./svc:
svc.yaml
~/gitlab/workspaces/test ❯ cat deploy/dep.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: test
spec:
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: test
spec:
containers:
- image: nginx:latest
name: nginx
~/gitlab/workspaces/test ❯ cat svc/svc.yaml
apiVersion: v1
kind: Service
metadata:
name: test
labels:
app: test
spec:
ports:
- name: http
port: 8080
targetPort: 8080
selector:
app: test
type: ClusterIP
~/gitlab/workspaces/test ❯ kube-linter lint . --do-not-auto-add-defaults --include dangling-service
KubeLinter 0.2.1-0-gc53952b8e3
svc/svc.yaml: (object: <no namespace>/test /v1, Kind=Service) no pods found matching service labels (map[app:test]) (check: dangling-service, remediation: Make sure your service's selector correctly matches the labels on one of your deployments.)
Error: found 1 lint errors
Coincidentally, I found through further testing that an issue we have with service accounts being inaccurately reported as non-existent also is resolved by having our service account definitions in the same directory as the deployment. It seems there is some bug with parsing the subdirectories.
We would like to keep our directory structure. Please advise how the presence of subdirectories causes interdependent checks (sa <-> deployment <-> service) to fail?
Aha! Yes, that is indeed working as expected. Your use-case is valid, though, so let's figure out how to solve for it.
Some background: kube-linter currently assumes that all YAMLs in a directory are part of the same app, and that YAMLs not in the same directory are not. In kube-linter terminology, each app is called a context. This explains the behaviour you are seeing.
The intention behind this design is to support running kube-linter lint . at the root of a git repository, and have it do all the work that's needed automatically, much like most code linters work. This means that we are supporting directory structures like the following, where multiple independent applications coexist in a repository:
Example 1:
my-git-root/
my-first-app/
my-first-dep.yaml
my-first-svc.yaml
my-second-app/
my-second-dep.yaml
my-second-svc.yaml
(For context, this is the approach we have used for storing configurations for our own internal services.)
Obviously, the weakness of this is that it doesn't handle use-cases like yours. Fundamentally, the problem is that we can't differentiate between a directory structure like yours, and one like the example above.
Whatever solution we design will have to handle the above case, as well as the following two cases:
Example 2:
my-git-root/
my-app/
deps/
dep.yaml
svc/
svc.yaml
Example 3:
my-git-root/
my-first-app/
deps/
dep.yaml
svc/
svc.yaml
my-second-app/
my-second-dep.yaml
my-second-svc.yaml
I can think of a couple of different solutions, and I would be very interested in your opinion here:
- Allow users to create a file (
.kube-linterrc, say) in a directory, which allows them to specify (in a to-be-determined format) that all YAML files in this directory, recursively, should be treated as belonging to the same context. (Downside: requires users to create another file, which is some friction.) - Add a flag to
kube-linter lintthat instructskube-linterto treat files discovered as belonging to the same context. (Downside: makes it impossible to handle cases like example 3 correctly, without multiplekube-linter lintinvocations).
Curious to hear any other ideas you have as well, as well as whether you have a preference between the two.
Hey Viswajith, yes I agree this is an interesting situation to have to account for.
I don't think option 1 will work in our case, as its an extra step to add that file to numerous pipelines.
Option 2, as you mentioned doesn't handle example 3, and we would want to accommodate other use cases, as we would hope others would do for us, so I have thought of another possibility and need your insight.
What can be done with annotations by kube-linter?
Could a new annotation be added that does as described in option 2, without the downsides of option 2?
The ignore-check.kube-linter.io/check-name syntax is great for ignoring checks, but if we want to hard code the context as one of the 3 options, could a new annotation be created to handle that?
Hmm, what would that look like in this case? It feels like that would require a lot more effort than option 1? To clarify, with option 1, I'm envisioning that the config file would be checked with to your code, and live and be maintained along with your Kube YAMLs.
You are right. It was just a thought.
Option 1 is the most flexible, as we can either check the file in, or create it on the fly in the pipeline when necessary. This would be fine.
Hi, hope you had a nice weekend. I wanted to follow up in order to ask if there has been any progress on this?
Thanks in advance.
Hi @tspearconquest, sorry, there hasn't. I expect to be done with this by the end of this week.
Hi @tspearconquest did you come up with a workaround for this? We are currently facing the same situation.
No, I'm afraid not. We continued with adding the annotation for now.