OPA Filters: Move Startup / Bundle Download out of Filter Creation

Is your feature request related to a problem? Please describe. The Open Policy Agent based filters today download bundles and fire up a virtual OPA instance during filter creation. This challenges some implicit assumptions in the code base that filter creation is a very fast and quick operation. Blocking filter creation for too long can block route processing which can cause unavailability of other routes and not update the endpoints that are targeted.

While there have been improvements to fail fast on bundle download errors, network delays and big bundles remain a lingering issue. In steady state we currently see no problem but this should be addressed.

Describe the solution you would like

We have discussed doing the following changes to make filter creation fast and not depend on external factors like network performance or bundle size:

Implement a route preprocessor that

• On first use (Skipper startup): Pre-Loads all OPA instances via the OPA Registry across all routes and waits until they have successfully finished or failed permanently
• On sub-sequent use: pre-load new OPA instances when they are detected but do not block the preprocessing further

Filter creation will be changed such that:

• It will lookup up if a ready instance can be obtained from the OPA Registry
• If none is found (not ready yet, errored out), it will just error out
• This will render the route erroneous and basically means that traffic cannot be served.

A note on CPU consumption

• On the initial start we can parallelize starting the OPA instances and bundle downloads because we want to start fast and the CPU consumption cannot have an effect on other I/O bound requests
• For the sub-sequent processing, we need to be careful to load new OPA instances in sequence to not overconsume CPU. This should ideally be combined if the central control loop for OPA bundle updates is enabled.

Additional context (optional) An initial skeleton for the implementation can be found here: https://github.com/zalando/skipper/compare/filters/openpolicyagent/initial-bundle-download

Would you like to work on it? Yes