OPA gatekeeper audit externaldata error: dial tcp: connect: cannot assign requested address

[mannbiher]

What happened?

I am using ratify to verify image signature using OPA gatekeeper external data. Ratify chart is installed following the documentation. After running for few days, OPA gatekeeper audit controller cannot open any new connections to ratify and ratifyconstraint would contain below error in violations.

- enforcementAction: warn
    group: ""
    kind: Pod
    message: 'System error calling external data provider: failed to send external
      data request: Post "https://ratify.gatekeeper-system:6001/ratify/gatekeeper/v1/verify":
      dial tcp 172.20.146.228:6001: connect: cannot assign requested address'
    name: abc-767bb47d54-kvb79
    namespace: abc
    version: v1

What should happen?

ratifyconstraint should show the actual violations.

Versions

Kubernetes version 1.27 (EKS) OPA gatekeeper 3.15.0 Ratify 1.1.0

Analysis

I looked at the external data provider code and can see it is happening because a new client is created for every request. There is no IdleConnTimeout set on transport so the old connections remain open. At some point of time no new connections can be opened and we get above error. Client should be created once for an external data provider and reused. It should also have defaults to limit concurrent idle connections e.g. setting values for MaxIdleConnsPerHost, IdleConnTimeout.

https://github.com/open-policy-agent/frameworks/blob/master/constraint/pkg/externaldata/request.go#L104

Go also recommends reuse of Client and transport.