emissary
emissary copied to clipboard
Extremely Slow response through emissary-ingress (+5 seconds)
Describe the bug
The response through emissary-ingress is very slow adding several seconds (consistently an extra 5 seconds) to each connection.
To Reproduce Steps to reproduce the behavior:
-
Install application that uses HTTP, for example, Dgraph on GKE:
helm repo add dgraph https://charts.dgraph.io && helm update # Install Dgraph helm install dg dgraph/dgraph \ --namespace dgraph \ --create-namespace \ --values - <<EOF zero: persistence: storageClass: premium-rwo size: 10Gi alpha: configFile: config.yaml: | security: whitelist: "0.0.0.0/0" persistence: storageClass: premium-rwo size: 30Gi EOF
-
Add appropriate CRDs for Dgraph
--- apiVersion: getambassador.io/v3alpha1 kind: Host metadata: name: dg-dgraph spec: hostname: "*" requestPolicy: insecure: action: Route --- apiVersion: getambassador.io/v3alpha1 kind: Listener metadata: name: dg-dgraph spec: port: 8080 protocol: HTTP securityModel: INSECURE hostBinding: namespace: from: SELF --- apiVersion: getambassador.io/v3alpha1 kind: Mapping metadata: name: dg-dgraph-http spec: hostname: dgraph.local prefix: / service: dg-dgraph-alpha.dgraph:8080 --- apiVersion: getambassador.io/v3alpha1 kind: Mapping metadata: name: dg-dgraph-grpc spec: hostname: "*" prefix: /api.Dgraph/ rewrite: /api.Dgraph/ service: dg-dgraph-alpha.dgraph:9080 grpc: True
-
Either update DNS record or /etc/hosts with emissary-ingress LB address
export SERVICE_IP=$(kubectl get svc emissary-ingress \ --namespace emissary \ --output jsonpath='{.status.loadBalancer.ingress[0].ip}' ) sudo sh -c "echo ${SERVICE_IP} dgraph.local" >> /etc/hosts"
-
Attempt to reach the service through emissary-ingress
DGRAPH_HOSTNAME_HTTP="dgraph.local" time curl -s http://$DGRAPH_HOSTNAME_HTTP/state | jq
Expected behavior
I expected that the time not add 5 seconds to the response. I am getting this from time:
real 0m5.327s
user 0m0.034s
sys 0m0.020s
Just to demonstrate that it is not the service itself, I can exec to a pod within the Dgraph cluster, and curl the service proxy and get a fast response:
# exec into container
kubectl exec -ti dg-dgraph-zero-0 --namespace dgraph -- bash
# inside container
export DGRAPH_HOSTNAME_HTTP=dg-dgraph-alpha-headless.dgraph.svc:8080
time curl -s http://$DGRAPH_HOSTNAME_HTTP/state | jq
From this, you can see the time result is:
real 0m0.066s
user 0m0.060s
sys 0m0.008s
Versions (please complete the following information):
- Ambassador: 3.9.1
- Kubernetes environment GKE v1.27.3-gke.100
Additional context
- GKE was provisioned using this guide: https://joachim8675309.medium.com/ultimate-baseline-gke-cluster-261c1b5544be
- Emissary-Ingress was installed using helm: https://www.getambassador.io/docs/emissary/latest/tutorials/getting-started
- Dgraph was deployed using the following below, but any web service should illustrate the problem.