Inference Extension: Secure communication with EPP

[sjberman]

trafficstars

As a platform owner, I want NGF to securely connect to the EPP using TLS, so that communication between NGINX and EPP is encrypted and authenticated.

Some investigation is required to see if the EPP has any documentation or ability to install with certificates. This could determine how we approach this.

Acceptance Criteria:

• TLS is used for communication between the Go application and EPP
• If the EPP has a documented or configurable way to install default certificates, see how we can integrate with that. See this comment Otherwise:
• By default, certificates should be provisioned by our control plane (using similar logic that our cert-generator Job does) and mounted to both the Go application and the EPP
• Recommended approach for production environments is to use cert-manager; this process should be tested and verified

Dev notes:

https://docs.nginx.com/nginx-gateway-fabric/install/secure-certificates/ is the guide where we talk about how to use cert-manager for control-plane -> agent communication, so we can likely do something similar here too.

Design doc: https://github.com/nginx/nginx-gateway-fabric/blob/main/docs/proposals/gateway-inference-extension.md

[bjee19]

After further investigation, we've concluded that this story is not currently implementable.

Essentially, because we don't provision the EPP and can't really modify it (the kubernetes administrators need to set it up), for us to mount a cert on the EPP, the implementation must provide a way. The EPP does provide a way for users to specify a certPath however, there is no way to actually mount a cert on the container through a volume/volumemount. So we can't mount a cert on the EPP, and thus can't setup that tls connection (current default is for a self-signed cert to be generated on the EPP and for implementations to skip ca validation). This is a known issue with this issue tracking further tls support.

This makes sense as to why all the other implementations do not have secure tls. Moving forwards our implementation will use an insecure connection (consistent with other implementations), which should be okay because this feature is hidden behind a feature flag, the gateway inference implementation is in ALPHA, and specifies that this should not be used in production.

When the gateway inference projection adds more support for tls, we should have a better path forwards in this.

A possible path forwards to consider is making a feature request to allow for volume/volumemount additions in the EPP helm chart. This would allow a user to deploy the EPP with an NGF recommended secret name containing the crt/key, however this would mean the EPP would not be able to function standalone and would require NGF to exist prior and have a secret deployed before the EPP is started, which currently goes against the standard getting started workflow.

This issue is blocked until the gateway inference issues listed above have progressed. This issue should not be a blocker to the completion of the inference extension work as this is not required to meet the API's core specifications.