azure-policy
azure-policy copied to clipboard
Azure
ContainerAllowedCapabilities: Adding any capability to allowedCapabilities list, makes it available to be used by ALL containers
Details of the scenario you tried and the problem that is occurring
We have an application container that needs SYS_PTRACE capability to be able to send SIGHUP signal to another container process. One way to allow this is by defining allowedCapabilities = ["SYS_PTRACE"] parameter. But if we do this, it basically means any other application container will also be able to use SYS_PTRACE capability, which is not intended. Other way is to exclude the app container that needs the capability by defining excludedContainers = ["app1"] as an example and not add anything to allowedCapabilities. This way no other app container can make use of SYS_PTRACE here. But that would allow app1 container to use any other capability (as its excluded from policy evaluation)!
Verbose logs showing the problem
Suggested solution to the issue
So I would like to suggest to have a parameter like allowedCapabilities as a map(/object) of array instead of plain array, where key will be container_name and value would be an array of capabilities allowed for that specific container. So we can specify it something like below, where app1 and app2 are container names allowedCapabilities = { "app1" : ["SYS_PTRACE"] , "app2" : ["SYS_RAWIO"] }
This way we can make sure that specific containers have the required capabilities they need.
Though I took ContainerAllowedCapabilities as an example, similar logic shall apply for few other policies like AllowedHostPaths.
If policy is Guest Configuration - details about target node
@fseldow will be great if you could take this to your team as an enhancement request, if that makes sense to you.
@fseldow will be great if you could take this to your team as an enhancement request, if that makes sense to you.
Do you need any more information from my side for this request to be considered ?
Hi @AshutoshNirkhe, do you mean you are looking for the functionality to change the capabilities that are allowed based on the container being evaluated?
Hi @AshutoshNirkhe, do you mean you are looking for the functionality to change the capabilities that are allowed based on the container being evaluated?
Hi @nehakulkarni123 thanks for getting back on this. Yes, so basically when we allow any capability, we should be able to allow that to only specific containers and not to all the containers by default.
Hi @AshutoshNirkhe, do you mean you are looking for the functionality to change the capabilities that are allowed based on the container being evaluated?
Hi @nehakulkarni123 thanks for getting back on this. Yes, so basically when we allow any capability, we should be able to allow that to only specific containers and not to all the containers by default.
@nehakulkarni123 could you please check if this request is good enough to be considered as IMO it will benefit fine tuning Policies exceptions more granularly that it's now ?
Hi @AshutoshNirkhe , sorry for late response. Here is another workaround is to assign multiple policies. For example, policy1: allowedCapabilites: [""], excludedContainers: ["app1", "app2"] policy2: allowedCapabilites: ["SYS_PTRACE"], labelSelector: {{ label of app1 }} policy3: allowedCapabilites: ["SYS_RAWIO"], labelSelector: {{ label of app2 }}
About the suggestion of new parameter, I cannot decide if it is suitable. So I create one issue in gatekeeper-library to have further discuss. That repo has more people specialize in constraint template policies design. https://github.com/open-policy-agent/gatekeeper-library/issues/178
Hi @AshutoshNirkhe , sorry for late response. Here is another workaround is to assign multiple policies. For example, policy1: allowedCapabilites: [""], excludedContainers: ["app1", "app2"] policy2: allowedCapabilites: ["SYS_PTRACE"], labelSelector: {{ label of app1 }} policy3: allowedCapabilites: ["SYS_RAWIO"], labelSelector: {{ label of app2 }}
About the suggestion of new parameter, I cannot decide if it is suitable. So I create one issue in gatekeeper-library to have further discuss. That repo has more people specialize in constraint template policies design. open-policy-agent/gatekeeper-library#178
@fseldow I didn't get what do you mean by multiple policies here ? I am referring to same AllowedCapabilities policy. Do you mean I can repeat same policy with different configurations ?
Correct. One policy definition can be assigned multiple times with different parameter values.
Correct. One policy definition can be assigned multiple times with different parameter values.
Sounds like that will work, but we will basically end up using new copy of policy for every new exception of container (though they shouldn't be many).
Correct. One policy definition can be assigned multiple times with different parameter values.
Sounds like that will work, but we will basically end up using new copy of policy for every new exception of container (though they shouldn't be many).
Just wanted to close on the loop. I ended up using the strategy of using multiple copies of same policy with different parameters. Want to confirm if that's the way going forward or there are any better mechanisms in place by now as I will be applying policies to another set of K8s clusters ?
Hi @fseldow so we hit another issue now. We have below situation,
policy1: allowedCapabilites: [""], excludedContainers: ["app1", "app2"] policy1a: allowedCapabilites: ["SYS_PTRACE"], labelSelector: {{ label of app1 }} policy1b: allowedCapabilites: ["NET_ADMIN"] }
So the policy1b is like 'NET_ADMIN' capability should be allowed for (almost) all pods whose container name will be say app2 but because those will be controlled by users, we can't restrict them to have a set of labels (to be used in labelSelector).
What should we do in such scenario ?
Hi @AshutoshNirkhe recently we added one more parameter named imageExclusion. It will use image tag prefix to exclude container. for example ["myregistry.acr.io/repo/image:*"] This parameter is more recommended than name-based exclusion. Will it work for you?
Hi @AshutoshNirkhe recently we added one more parameter named imageExclusion. It will use image tag prefix to exclude container. for example
["myregistry.acr.io/repo/image:*"]This parameter is more recommended than name-based exclusion. Will it work for you?
@fseldow I think that's alternative to excludedContainers, but shall serve the same purpose. We ended up doing this hack,
policy1b: { allowedCapabilites: ["NET_ADMIN"], excludedContainers: ["app1"] }
But that basically selects everything else apart from app1.
- The benefit of image exclusion is that name-based exclusion can be easily bypassed. And it will no longer has restriction on application name unify.
- Since you are using multiple policies combination, policy initiative might be a long time solution to help you maintain. The benefit is that you can define custom defined parameter (initiative) and then define logic to simply parse your initiative parameter to these sub policy parameter value. Below is an example of initiative json file with my idea. I have not tested it. But you can refer it. The main point is that the initiative parameter is changed to allow a list of images for certain capabilities.
{
"properties": {
"displayName": "xxx",
"policyType": "Custom",
"description": "xxx",
"metadata": {
"version": "1.0.0",
"category": "Kubernetes"
},
"parameters": {
"effect": {
"type": "String",
"metadata": {
"displayName": "Effect",
"description": "'Audit' allows a non-compliant resource to be created or updated, but flags it as non-compliant. 'Deny' blocks the non-compliant resource creation or update. 'Disabled' turns off the policy."
},
"allowedValues": [
"Audit",
"Deny",
"Disabled"
],
"defaultValue": "Audit"
},
"allowedCapabilities": {
"type": "Array",
"metadata": {
"displayName": "common allowed capacity",
"description": "It is for the strictest allowed list"
},
"defaultValue": [
]
},
"imageAllowedForSYS_PTRACE": {
"type": "Array",
"metadata": {
"displayName": "imageAllowedForSYS_PTRACE",
"description": "imageAllowedForSYS_PTRACE"
},
"defaultValue": [
"image1"
]
},
"imageAllowedForSYS_RAWIO": {
"type": "Array",
"metadata": {
"displayName": "imageAllowedForSYS_RAWIO",
"description": "imageAllowedForSYS_RAWIO"
},
"defaultValue": [
"image2"
]
},
"imageAllowedForSYS_ADMIN": {
"type": "Array",
"metadata": {
"displayName": "imageAllowedForSYS_ADMIN",
"description": "imageAllowedForSYS_ADMIN"
},
"defaultValue": [
"image3"
]
}
},
"policyDefinitions": [
{
"policyDefinitionId": "/providers/Microsoft.Authorization/policyDefinitions/c26596ff-4d70-4e6a-9a30-c2506bd2f80c",
"policyDefinitionReferenceId": "policybase",
"parameters": {
"effect": {
"value": "[parameters('effect')]"
},
"allowedCapabilities": {
"value": "[parameters('allowedCapabilities')]"
},
"excludedImages": {
"value": "[union(parameters('imageAllowedForSYS_PTRACE'), parameters('imageAllowedForSYS_RAWIO'), parameters('imageAllowedForSYS_ADMIN'))]"
}
}
},
{
"policyDefinitionId": "/providers/Microsoft.Authorization/policyDefinitions/c26596ff-4d70-4e6a-9a30-c2506bd2f80c",
"policyDefinitionReferenceId": "policySYS_PTRACE",
"parameters": {
"effect": {
"value": "[parameters('effect')]"
},
"allowedCapabilities": {
"value": ["SYS_PTRACE"]
},
"excludedImages": {
"value": "[union(parameters('imageAllowedForSYS_RAWIO'), parameters('imageAllowedForSYS_ADMIN'))]"
}
}
},
{
"policyDefinitionId": "/providers/Microsoft.Authorization/policyDefinitions/c26596ff-4d70-4e6a-9a30-c2506bd2f80c",
"policyDefinitionReferenceId": "policySYS_RAWIO",
"parameters": {
"effect": {
"value": "[parameters('effect')]"
},
"allowedCapabilities": {
"value": ["SYS_RAWIO"]
},
"excludedImages": {
"value": "[union(parameters('imageAllowedForSYS_PTRACE'), parameters('imageAllowedForSYS_ADMIN'))]"
}
}
},
{
"policyDefinitionId": "/providers/Microsoft.Authorization/policyDefinitions/c26596ff-4d70-4e6a-9a30-c2506bd2f80c",
"policyDefinitionReferenceId": "policySYS_ADMIN",
"parameters": {
"effect": {
"value": "[parameters('effect')]"
},
"allowedCapabilities": {
"value": ["SYS_ADMIN"]
},
"excludedImages": {
"value": "[union(parameters('imageAllowedForSYS_PTRACE'), parameters('imageAllowedForSYS_RAWIO'))]"
}
}
}
]
},
"id": "/providers/Microsoft.Authorization/policySetDefinitions/xxxxxxxxxxxxxxx",
"name": "xxxxxxxxxxxxxxxxx"
}
Thanks @fseldow but we didn't want to exclude things (be it container name or images) for the specific policy copies (e.g. policybase doesn't allow any capabilities but excludes all next policy copies. Then policySYS_RAWIO needs to allow only those apps that need SYS_RAWIO capability and SYS_ADMIN needs to allow only those apps that need SYS_ADMIN etc). So we need something includeName or includeImage to select that particular container or image. We can't use labelSelectors because different apps are going to add those containers under their pod spec and we can't control that.
Sorry @AshutoshNirkhe, currently we do not have plan to add includeImage or includeName. For my personal view, policy should be a general standard. Allowed capabilities mean the list we trust. If certain image need to exclude for another capability, the image itself should be evulated as safe enough and no longer need additional policy to monitor.
As a workaround, the initiative I provided at option 2 (in fact same logic as your current workaround but more easy to manage) is to realize the the inclusion parameter. You can find new parameters are imageAllowedForSYS_ADMIN
Sorry again for the inconvenience.