Feature request: Support network policies

[olljanat]

As Acorn can be used to deploy multiple containers as bundle it it would be nice to have also option to generate network policy which isolates application from others.

[ibuildthecloud]

Yes, this absolutely will come. The design of acorn is such that all network communication paths are known and have some logical boundary. We will either be generating network policies and/or service mesh config (istio/linkerd) to enforce this. This is key future feature for the security posture of Acorn. One of the great things about Acorn is we know so much about the application behavior that we can create good implicit rules.

[ibuildthecloud]

Researching this a bit, network policies would be extremely simple. LInkerd looks straight forward too and Istio is rocket science like normal. So we will start with NetworkPolicies and go from there.

[olljanat]

In case of one app with multiple containers it should be very simple yes. In case of multiple apps you will need also figure out some nice way to refer another app in Acornfile bur perhaps even that can be just name reference.

[g-linville]

Design Document

Requirements

• All ingress traffic to a project is blocked by default, but all egress traffic is allowed
• Traffic to published ports is allowed from all sources

UX and API changes

• New flag for acorn install that disables the creation of NetworkPolicies (something like --disable-network-policies).
  • This might also result in a slight API change for the struct that handles these installation settings. I haven't looked at this part of the code yet.

Visualization

Additional considerations

• Rather than allowing all traffic to published ports, would it be better to simply allow traffic from the ingress controller namespace? That would probably be a better design, but it would require the user to pass in the name of the namespace where the ingress controller is installed (probably an argument to acorn install). The reason we might want to do this is because a lot of people configure their ingress controllers with TLS, and might want services within the cluster that consume other ingress-exposed services to go out of the cluster and back in through the ingress controller rather than just connecting directly to the ingress-exposed services and bypassing TLS.
• Backwards compatibility: if we turn on NetworkPolicies by default (and I think we should), this has the potential to break some users' existing configurations, if they were relying on applications in separate projects to be able to talk to each other on non-published ports. We would definitely need to warn users about this.
• Tests: I wrote a unit and integration test for this.
• Time estimate: I think this should be a fairly easy feature to add that we can easily land in the next release.

[cjellick]

New flag for acorn install that disables the creation of NetworkPolicies (something like --disable-network-policies).

I am wondering if we should we have it disabled by default and you need to explicitly enable it?

Edit: for "secure by default" we should turn it on by default, but im worried about adding unnecessary friction for local developers. Of course, assuming they had a working net policy enforcer locally, maybe we'd be doing them a favor by having it on by default because that wouldn't give them bad assumptions about how their apps will run in prod.

Traffic between different projects is blocked by default

To nitpick this, I think it is more accurate to say that all ingress traffic to a project should be blocked. All egress allowed

[cjellick]

Publishing a port puts it on an ingress. thatll def make it publicly accessible. That makes sense to me.

Should exposing a port poke a hole for it in the network policy so that it can be reached by other workloads in the cluster?

Edit: I guess this is related to one of your questions:

Rather than allowing all traffic to published ports, would it be better to simply allow traffic from the ingress controller namespace?

When we've brought up publishing, exposing, and just defining ports before, I get fairly confused about the exact expected behavior of each and I think @ibuildthecloud says its murky and needs refactor. I think they need to be really well defined WRT net policy.

[g-linville]

Traffic between different projects is blocked by default

To nitpick this, I think it is more accurate to say that all ingress traffic to a project should be blocked. All egress allowed

Yeah that is definitely a better way of phrasing it.

Should exposing a port poke a hole for it in the network policy so that it can be reached by other workloads in the cluster?

Yeah I'm not sure about this. My current implementation does poke this hole, but like I suggested, it might be better to have the user provide the namespace where the ingress controller lives, and allow access only from that namespace (and maybe also a label selector that selects only the ingress controller pods).

[cjellick]

Not sure user specifying ingress namespace will suffice, particularly with the expose keyword (which doesnt involve ingress at all) and is explicitly defined as making a workload to other services in the cluster: https://docs.acorn.io/running/networking#expose-individual-ports

[g-linville]

Not sure user specifying ingress namespace will suffice, particularly with the expose keyword (which doesnt involve ingress at all) and is explicitly defined as making a workload to other services in the cluster: https://docs.acorn.io/running/networking#expose-individual-ports

For expose I figured that just meant exposed to other services within the same project. So with NetPols, I think we can treat internal and expose ports the same way.

[g-linville]

This is what we decided on (for the time being): published and exposed ports will both have NetPols created for them that allow all ingress traffic.

[cjellick]

moving this into the v0.7.0 milestone

[cjellick]

moving this into the v0.7.0 milestone

[sangee2004]

Tested with acorn version v0.6.0-21-gb1b07672+b1b07672

On Acorn install , Network policy is enabled by default - disableNetworkPolicies: false Project isolation for ping, http and ssh access works as expected . project1 - App1 - containers - web and ubuntu project2 - App2 - containers - web and ubuntu and TestContainerApp - containers - testcon

From testcon

1. Able to curl App2.web's pod ip

2. Able to ssh App2.web's ubuntu ip

3. Able to ping App2.web's pod ip and App1.web's ubuntu ip

4. Not able to curl App1.web's pod ip

root@myapp:/app# curl 10.42.0.91    
curl: (7) Failed to connect to 10.42.0.91 port 80: Connection refused

5. Not able to ssh App1.web's ubuntu ip

root@myapp:/app# ssh [email protected]
ssh: connect to host 10.42.0.90 port 22: Connection refused

6. Not able to ping App1.web's pod ip and App1.web's ubuntu ip

From 10.42.0.1 icmp_seq=1 Destination Port Unreachable
From 10.42.0.1 icmp_seq=2 Destination Port Unreachable
From 10.42.0.1 icmp_seq=3 Destination Port Unreachable

From Ingress end points, I am not able to access ingress endpoint from other containers with in the same app , from containers with in the same project and from containers from different projects . I am able to access ingress endpoint only from with in the same container that published the ingress.

@g-linville Could you please confirm the behavior seen for Ingress endpoints ?

[g-linville]

From Ingress end points, I am not able to access ingress endpoint from other containers with in the same app , from containers with in the same project and from containers from different projects . I am able to access ingress endpoint only from with in the same container that published the ingress.

@g-linville Could you please confirm the behavior seen for Ingress endpoints ?

@sangee2004 In local clusters, this is how it should behave, yeah. The reason for this is that all of the *.local.on-acorn.io domains resolve to 127.0.0.1 (localhost). So it won't work for trying to connect across pods.

In non-local environments I think you should be able to reach another pod's ingress endpoint from inside a different pod, but I haven't been able to test that yet.

[sangee2004]

Tested with acorn version v0.6.0-113-g83b7b9f3+83b7b9f3

On Acorn install , Network policy is enabled by default - networkPolicies: true Project isolation for ping, http and ssh access works as expected . project1 - App1 - containers - web and ubuntu project2 - App2 - containers - web and ubuntu and TestContainerApp - containers - testcon

From testcon

1. Able to curl App2.web's pod ip

2. Able to ssh App2.web's ubuntu ip

3. Able to ping App2.web's pod ip and App1.web's ubuntu ip

4. Not able to curl App1.web's pod ip

root@myapp:/app# curl 10.42.0.91    
curl: (7) Failed to connect to 10.42.0.91 port 80: Connection refused

5. Not able to ssh App1.web's ubuntu ip

root@myapp:/app# ssh [email protected]
ssh: connect to host 10.42.0.90 port 22: Connection refused

6. Not able to ping App1.web's pod ip and App1.web's ubuntu ip

From 10.42.0.1 icmp_seq=1 Destination Port Unreachable
From 10.42.0.1 icmp_seq=2 Destination Port Unreachable
From 10.42.0.1 icmp_seq=3 Destination Port Unreachable

[sangee2004]

When containers have published ports , then we are able to reach (http access) this container cross project using podip. This is as expected.

Having ingress-controller-namespace set - acorn install --ingress-controller-namespace=kube-system will result in containers that have published ports , to be not reachable (http access) cross project using podip.