THREESCALE-8508 - /admin/api/account/proxy_configs endpoint for configuration loading

[samugi]

This PR uses /admin/api/account/proxy_configs for the standard configuration loading process, as the default endpoint that APIcast will attempt to fetch the configuration from when the portal endpoint is provided without a path (admin portal endpoint).

With this proposed implementation, APICAST_LOAD_SERVICES_WHEN_NEEDED is deprecated and the configuration is fetched "when needed" by default whenever possible, with the following logic:

• THREESCALE_PORTAL_ENDPOINT has no path
  • APICAST_CONFIGURATION_LOADER=boot
    • http://${THREESCALE_PORTAL_ENDPOINT}/admin/api/account/proxy_configs/<env>.json?version=version (i.e. all services from proxy_configs endpoint). Using the new endpoint now one request is enough, no need to loop through all services.
  • APICAST_CONFIGURATION_LOADER=lazy
    • http://${THREESCALE_PORTAL_ENDPOINT}/admin/api/account/proxy_configs/<env>.json?host=host&version=version (i.e. services that match host from proxy_configs endpoint). This matches the current behaviour when APICAST_LOAD_SERVICES_WHEN_NEEDED is enabled.
• THREESCALE_PORTAL_ENDPOINT has a path
  • APICAST_CONFIGURATION_LOADER=boot
    • http://${THREESCALE_PORTAL_ENDPOINT}/<env>.json (i.e. all services from master endpoint). Same as the current behaviour.
  • APICAST_CONFIGURATION_LOADER=lazy
    • http://${THREESCALE_PORTAL_ENDPOINT}/<env>.json?host=host (i.e. services that match host from master endpoint). Same as the current behaviour.

i.e. the path appended to ${THREESCALE_PORTAL_ENDPOINT} is:

	APICAST_CONFIGURATION_LOADER=boot	APICAST_CONFIGURATION_LOADER=lazy
endpoint has no path	/admin/api/account/proxy_configs/${env}.json?version=version	/admin/api/account/proxy_configs/${env}.json?host=host&version=version
endpoint has a path	/${env}.json	/${env}.json?host=host

Update: the only exception is when the environment variable APICAST_SERVICE_%s_CONFIGURATION_VERSION is configured: this is not compatible with either of the endpoints above and will just not work when THREESCALE_PORTAL_ENDPOINT points to master (same as current behaviour) while it requires falling back to the original logic and loading services one by one when pointing to the admin portal.

[eguzki]

Code looks ok. Also tests.

Just to verify: APIcast no longer will request proxy configuration for a given service ID. When "lazy" configuration, the endpoint will have a query param for the given "host". Right?

Question: what if two services have the same host?

[eguzki]

For managed APIcast deployment (on-prem releases), the THREESCALE_PORTAL_ENDPOINT env var has path configured to /master/api/proxy/configs https://github.com/3scale/3scale-operator/blob/69c1e2c41b2b9d308278e513d07d405b5d8196e6/pkg/3scale/amp/component/system_options.go#L114

This is still valid or should be changed?

[samugi]

Code looks ok. Also tests.

Just to verify: APIcast no longer will request proxy configuration for a given service ID. When "lazy" configuration, the endpoint will have a query param for the given "host". Right?

Question: what if two services have the same host?

That's right. If multiple services have the same host they will all be loaded in APIcast at once and treated as usual.

[samugi]

Code looks ok. Also tests.

Just to verify: APIcast no longer will request proxy configuration for a given service ID. When "lazy" configuration, the endpoint will have a query param for the given "host". Right?

Question: what if two services have the same host?

That's fine @eguzki, since it points to master, this PR should have no effect on gateways configured that way.

[secretsurf]

Is this lost in the ether?

[kevprice83]

Is this lost in the ether?

No @secretsurf sorry I am supposed to be reviewing and testing this but too many other burning issues I didn't manage to get to this yet.

[eguzki]

One thing I concern about this is the paginated endpoint. From the description in the issue

Open Points:  

    Currently the endpoint /admin/api/account/proxy_configs/${ENV}

 is paginated, and APIcast does not handle pages, so we would need to 
make the pagination optional or to implement pagination on APIcast too before we can proceed with this

and I do not remember to see any pagination implementation. Maybe it was done after my initial review

[eguzki]

@pehala @thomasmaas this PR is ready for review

[eguzki]

leaving pagination implementation for a (soon) coming PR

[eguzki]

This seems like a breaking change, can you enable the old behaviour? Does enabling the old behaviour even make sense?

Basically, we are making an opt-in behavior the default one. Opt-in is no longer needed, hence removal of env var.

[pehala]

This seems like a breaking change, can you enable the old behaviour? Does enabling the old behaviour even make sense?

Basically, we are making an opt-in behavior the default one. Opt-in is no longer needed, hence removal of env var.

Right, and is there a option to opt-out? Does it even make sense to opt-out or is the new behaviour simply superior?

[eguzki]

This seems like a breaking change, can you enable the old behaviour? Does enabling the old behaviour even make sense?

Basically, we are making an opt-in behavior the default one. Opt-in is no longer needed, hence removal of env var.

Right, and is there a option to opt-out? Does it even make sense to opt-out or is the new behaviour simply superior?

For me it does not make sense to opt-out. But if for whatever reason there is a need to opt-out from the current default behavior and get all services and ask for proxy config for each service, there are ways.

• Use APICAST_SERVICE_${SERVICE_ID}_CONFIGURATION_VERSION env var with a not existing ${SERVICE_ID}
• ~Use APICAST_SERVICES_FILTER_BY_URL with a filter that always match.~