Applications breaking after applying auto discovered kubearmor policies

[salman-accuknox]

General Information

• Environment: GKE cluster (Image type: Container-Optimized OS with containerd (cos_containerd))
• accuknox-cli and KubeArmor version:

➜  accuknox version
accuknox-cli version 0.1.14 linux/amd64 BuildDate=2022-07-22T11:27:46Z
current version is the latest
kubearmor image (running) version kubearmor/kubearmor:stable
karmor version 0.7.6 linux/amd64 BuildDate=2022-06-29T03:58:05Z
current version is the latest
kubearmor image (running) version kubearmor/kubearmor:stable

Issue Faced

• Applications breaking after applying discovered kubearmor policies.
• Tested with 2 applications.

Expected Output

• The applications should not break and work as expected.
• Application should work in the least privileged environment.

Steps to reproduce

• Deploy tweaked google microservice application in g-ms namespace

k apply -f https://raw.githubusercontent.com/accuknox/samples/main/microservice-demo/release/kubernetes-manifests.yaml -n g-ms

• Deploy wordpress application in wp-ms namespace

k apply -f https://raw.githubusercontent.com/accuknox/samples/main/wordpress-demo/k8s-wordpress.yaml -n wp-ms

• Discover policies for applications

accuknox port-forward discovery-engine

accuknox discover -n wp-ms -f yaml > wp-ms-ad.yaml

accuknox discover -n g-ms -f yaml > g-ms-ad.yaml

• Apply discovered policies

 k apply -f g-ms-ad.yaml

k apply -f wp-ms-ad.yaml

Screenshots and logs

• Wordpress application before applying policies(After initial setup)

• Wordpress application after applying policies

• kubearmor log when trying to access the webpage

➜  ~ accuknox log application --namespace wp-ms
gRPC server: localhost:32767
Created a gRPC client (localhost:32767)
Checked the liveness of the gRPC server
Started to watch alerts
== Alert / 2022-07-25 10:01:57.436704 ==
Cluster Name: default
Host Name: gke-cys-july24-default-pool-536afb90-6psr
Namespace Name: wp-ms
Pod Name: wordpress-7d5566b7b7-6wqg5
Container ID: 120ff7c36a45db89028d7bd900fab2a80308acaa4729c424c1799c6ac80574c3
Container Name: wordpress
Labels: tier=frontend,app=wordpress
Policy Name: DefaultPosture
Type: MatchedPolicy
Source: /usr/sbin/apache2
Operation: File
Resource: /var/www/html/.htaccess
Data: syscall=SYS_OPEN flags=O_RDONLY|O_CLOEXEC
Action: Block
Result: Permission denied
== Alert / 2022-07-25 10:01:57.868059 ==
Cluster Name: default
Host Name: gke-cys-july24-default-pool-536afb90-6psr
Namespace Name: wp-ms
Pod Name: wordpress-7d5566b7b7-6wqg5
Container ID: 120ff7c36a45db89028d7bd900fab2a80308acaa4729c424c1799c6ac80574c3
Container Name: wordpress
Labels: tier=frontend,app=wordpress
Policy Name: DefaultPosture
Type: MatchedPolicy
Source: /usr/sbin/apache2
Operation: File
Resource: /var/www/html/.htaccess
Data: syscall=SYS_OPEN flags=O_RDONLY|O_CLOEXEC
Action: Block
Result: Permission denied

• Google microservice application before applying policies (when clicking place order )

• After applying the policies

• KubeArmor logs

➜  ~ accuknox log application --namespace g-ms
gRPC server: localhost:32767
Created a gRPC client (localhost:32767)
Checked the liveness of the gRPC server
Started to watch alerts
== Alert / 2022-07-25 10:14:12.179483 ==
Cluster Name: default
Host Name: gke-cys-july24-default-pool-536afb90-fdn6
Namespace Name: g-ms
Pod Name: shippingservice-7d769946f7-4nmrd
Container ID: 1c30ed8ec164a81eef16adea1cd0147c494ba7b952099958fadc6754aa954d23
Container Name: server
Labels: app=shippingservice
Policy Name: DefaultPosture
Type: MatchedPolicy
Source: /usr/bin/runc
Operation: File
Resource: /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
Data: syscall=SYS_OPENAT fd=-100 flags=O_RDONLY
Action: Block
Result: Permission denied
== Alert / 2022-07-25 10:14:12.206555 ==
Cluster Name: default
Host Name: gke-cys-july24-default-pool-536afb90-fdn6
Namespace Name: g-ms
Pod Name: emailservice-6b66bc698c-xxx58
Container ID: 877e71b325e67dfd66fa1e8181a08bed4b0be1d84c5be47c42ba0039011dfda8
Container Name: server
Labels: app=emailservice
Policy Name: DefaultPosture
Type: MatchedPolicy
Source: /usr/bin/runc
Operation: File
Resource: /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
Data: syscall=SYS_OPENAT fd=-100 flags=O_RDONLY
Action: Block
Result: Permission denied
== Alert / 2022-07-25 10:14:12.277612 ==
Cluster Name: default
Host Name: gke-cys-july24-default-pool-536afb90-x0jj
Namespace Name: g-ms
Pod Name: checkoutservice-7b5ccb7fcb-7lxlv
Container ID: c0af18c403eb7932eaa114c39a7d3ed97a6f15ab2d1d9be32be775b06dd805e8
Container Name: server
Labels: app=checkoutservice
Policy Name: DefaultPosture
Type: MatchedPolicy
Source: /src/checkoutservice
Operation: Network
Resource: sa_family=AF_INET sin_port=53 sin_addr=10.92.0.10
Data: syscall=SYS_CONNECT fd=8
Action: Block
Result: Permission denied

• Note: In the google microservice app, there are 12 pods running. Auto discovery has given the policies for 11 pods. Policies with app: redis-cart were missing

Sysdump https://drive.google.com/file/d/1jAudEuxum7TwqwZEEp4M8YsrO5AuqZZw/view?usp=sharing