Proposal: support external name service

[tao12345666333]

I have seen some voices in the community, hoping that APISIX Ingress can proxy external services e.g: #813 #645

For these two types of requirements, it is a relatively simple requirement for #645, we only need to add the corresponding External name type service to complete.

But for #645 , I found a very interesting situation. No other Ingress controller implements similar functionality yet, and I think this would be a huge feature.

APISIX actually supports setting the domain name to nodes in the upstream. But APISIX Ingress is not yet supported.

To achieve the above function, we can set a special resolveGranularity to directly convert the record of the external name to Node.

[elvis-cai]

+1, multi external url support with active health check and upstream node pass is the major block before we rollout Apisix ingress controller in all of our kubernetes clusters. It’s quite common for an organisation to expose services internally or externally in hybrid environment without the boundaries of kubernetes cluster concept, you can easily have your apps running in serverless platform, kubernetes cluster in cloud and in bare metal server on your on-premise data center.

GCP is also actively developing managed istio control plane called traffic director, it can be deployed anywhere and utilise envoy or any xDS v3 compatible data plane to satisfy the requirement we have. Ambassador also has a native CRD resoruce mapping to define your service with external url directly https://www.getambassador.io/docs/edge-stack/latest/topics/using/intro-mappings/#introduction-to-the-mapping-resource, the missing part if active health check no sure if it’s supported or not .https://github.com/emissary-ingress/emissary/issues/3761 Gloo Edge also supports native CRD to configure multi url with active health check and upstream node name passing feature we needed, example config could be found here https://gist.github.com/elvis-cai/d6978b1113edb10359da1358146f61aa.

[gxthrj]

I think the feature is useful, If we can configure external name type service in ApisixUpstream resource as a node of upstream object in APISIX，we can use health check feature between external services.

[gxthrj]

GCP is also actively developing managed istio control plane called traffic director, it can be deployed anywhere and utilise envoy or any xDS v3 compatible data plane to satisfy the requirement we have. Ambassador also has a native CRD resoruce mapping to define your service with external url directly https://www.getambassador.io/docs/edge-stack/latest/topics/using/intro-mappings/#introduction-to-the-mapping-resource, the missing part if active health check no sure if it’s supported or not .https://github.com/emissary-ingress/emissary/issues/3761 Gloo Edge also supports native CRD to configure multi url with active health check and upstream node name passing feature we needed, example config could be found here https://gist.github.com/elvis-cai/d6978b1113edb10359da1358146f61aa.

Yes, The idea is generally the same, I think supporting the feature in ApisixUpstream resource as a native CRD is a good idea.

[tao12345666333]

In order for us to reach an agreement as soon as possible, I will give a more detailed design ASAP.

[PGDream]

we use scenes is multi k8s cluster，A and B service not in diff k8s cluster，A mirrors-flow to B now not support.

[tokers]

we use scenes is multi k8s cluster，A and B service not in diff k8s cluster，A mirrors-flow to B now not support.

Could you describe you problem accurately? What is "not in diff k8s cluster"? You want to mirror traffic for service A to service B?

[PGDream]

in different k8s cluster, A cluster、B cluster，A cluster a1 service call B cluster a2 service

[tokers]

in different k8s cluster, A cluster、B cluster，A cluster a1 service call B cluster a2 service

OK, I see, ExternalName is useful for this scenario.

[lingsamuel]

Hi, this is my proposal. Feel free to comment/review.

Background

The users' environment may be not a simple single K8s cluster, but may be multiple K8s clusters, a mix of K8s clusters and bare metal machines, or rely on third-party services. The current ApisixRoute does not support such complex environments.

It would be a useful feature if ApisixRoute supports using ExternalName Service or domain name as a backend directly.

Solution

APISIX does support setting domain names or IP addresses as upstream nodes. We can extend our CRD to support this scenario.

For external service, both the ExternalName service inside the K8s cluster and the third-party service outside the cluster will only have one node configured in the translated upstream nodes field, unlike the K8s service which may have several nodes. This allows multiple external services to be merged into a single upstream and gain healthCheck capabilities.

Design

We add a field named upstream to ApisixRoute, which has the same level as "backends". The user must fill in at least one of upstream or backends. The field upstream will refer to an ApisixUpstream CRD, where the external services are defined in the newly added field externalNode, indicating that the upstream node is pre-defined and does not need to resolve. An externalNode must have a name field pointing to the final node location, which can be a service name pointing to a k8s external name service, an IP address or a domain name. If it points to a k8s external name service, the type field should be service, or it should be the default value domain. The node can also be configured with an optional load balancing weight, which defaults to 100.

apiVersion: apisix.apache.org/v2 # v2 only
kind: ApisixRoute
metadata:
  name: external-example
spec:
  http:
  - name: example
    match:
      hosts:
      - external.example.com
      paths:
      - "/*"
    upstreams:
    - name: external-example # Refers to a v2 ApisixUpstream
      weight: 50
---
apiVersion: apisix.apache.org/v2 # v2 only
kind: ApisixUpstream
metadata:
  name: external-example
spec:
  externalNodes: # Indicates that this upstream doesn't need to resolve nodes
  - name: "httpbin.org"
    # could be "domain", "serivce"
    # if the value is `service`, the name must refers to a K8s ExternalName service
    type: domain
    weight: 100 # default value is 100
  healthCheck: # same as normal ApisixUpstream
    # ....

Using upstream and backends together

Users can use both upstream and backend. A route-level traffic-split plugin will be generated normally. Please note that since APISIX does not support health checks between multiple upstreams, we will only support health checks within the upstream.nodes. This means that the healthCheck section defined in ApisixUpstream is at the upstream level and its effect is limited to that upstream only, not at the Route level.

---

TODO:

• [x] Basic function: #1306
• [x] Add documentation
• [x] Support custom scheme and port

[tao12345666333]

SGTM

This method can ensure good compatibility and rich functions

[tao12345666333]

#1500

[tao12345666333]

#1527

[tao12345666333]

I think we can close this one . Thanks all.

• [x] #1306
• [x] #1500
• [x] #1527

[kindomLee]

Is externalNodes currently compatible with IP address with port? like:

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  name: test-upstream
spec:
  externalNodes:
  - type: Domain
    name: 10.0.10.21:5601

[tao12345666333]

@kindomLee no, you can file a new feature request issue.