apisix
apisix copied to clipboard
apache
bug: About the instability of Apisix's proxying of gRPC services.
Current Behavior
I am using apisix to proxy my grpc service, and I found the following issue while doing benchmark. This is the access log and the error rpc error: code = Unknown desc = unexpected HTTP status code received from server: 0 (); malformed header: missing HTTP content-type returned by the grpc client. This error is very frequent and seriously affects the use of our production environment.
The host is: ep-mzinkvxm.endpoint.windmill.com
here is my route config:
- name: grpc
priority: 1
match:
hosts:
- {{ .Release.Name }}.endpoint.windmill.com
paths:
- "/*"
backends:
- serviceName: {{ .Release.Name }}
servicePort: grpc
here is the config i copy from apisix-admin:
{
"uris": ["/*"],
"name": "default_ep-mzinkvxm_grpc",
"desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
"priority": 1,
"hosts": ["[ep-mzinkvxm.endpoint.windmill.com](http://ep-mzinkvxm.endpoint.windmill.com/)"],
"upstream_id": "c54210dc",
"labels": {
"managed-by": "apisix-ingress-controller"
},
"status": 1
}
and there is some error I found from apisix-ingress-controller, i don't know if this error is related to this phenomenon.
2023-03-29T20:35:08+08:00 warn ingress/apisix_route.go:440 sync ApisixRoute failed, will retry {"object": {"Type":1,"Object":{"Key":"default/ep-fuphptti","OldObject":null,"GroupVersion":"[apisix.apache.org/v2](http://apisix.apache.org/v2)"},"OldObject":null,"Tombstone":null}, "error": "6 errors occurred:\n\t* unexpected status code 503; error message: {\"error_msg\":\"has no healthy etcd endpoint available\"}\n\n\t* unexpected status code 503; error message: {\"error_msg\":\"has no healthy etcd endpoint available\"}\n\n\t* unexpected status code 503; error message: {\"error_msg\":\"has no healthy etcd endpoint available\"}\n\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [cf3c7922]: has no healthy etcd endpoint available\"}\n\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [b83b49b4]: has no healthy etcd endpoint available\"}\n\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [2132180e]: has no healthy etcd endpoint available\"}\n\n\n"}
Expected Behavior
I hope this mistake won't happen again.
Error Logs
No response
Steps to Reproduce
- Running APIsix on k8s using Helm Chart.
- Configure APIsix routes using YAML
{{- if .Values.ingress.enabled -}}
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
name: {{ .Release.Name }}
spec:
http:
- name: http
priority: 3
match:
paths:
- /{{ .Release.Name }}/http/*
backends:
- serviceName: {{ .Release.Name }}
servicePort: http
plugins:
- name: proxy-rewrite
enable: true
config:
regex_uri: ["^/{{ .Release.Name }}/http/(.*)", "/$1"]
- name: api-breaker
enable: true
config:
break_response_code: 502
unhealthy:
http_statuses: [500, 503]
failure: 3
healthy:
http_statuses: [200]
successes: 1
- name: grpc
priority: 1
match:
hosts:
- {{ .Release.Name }}.endpoint.windmill.com
paths:
- "/*"
backends:
- serviceName: {{ .Release.Name }}
servicePort: grpc
- name: metrics
priority: 2
match:
paths:
- /{{ .Release.Name }}/metrics/*
backends:
- serviceName: {{ .Release.Name }}
servicePort: metrics
plugins:
- name: proxy-rewrite
enable: true
config:
regex_uri: ["^/{{ .Release.Name }}/metrics/(.*)", "/$1"]
---
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
name: {{ .Release.Name }}
spec:
loadbalancer:
type: ewma
retries: {{ .Values.ingress.retries }}
timeout:
connect: {{ .Values.ingress.timeout.connect }}
send: {{ .Values.ingress.timeout.send }}
read: {{ .Values.ingress.timeout.read }}
portLevelSettings:
- port: 8000
scheme: http
- port: 8001
scheme: grpc
- port: 8002
scheme: http
{{- end }}
- use grpc client to request
Environment
-
APISIX version (run
apisix version):
-
Operating system (run
uname -a):Linux apisix-656c7b655c-dlqgk 4.14.0_1-0-0-50 #1 SMP Wed Jun 1 18:07:32 CST 2022 x86_64 Linux -
OpenResty / Nginx version (run
openresty -Vornginx -V):
-
etcd version, if relevant (run
curl http://127.0.0.1:9090/v1/server_info):3.5.4-debian-11-r14 -
APISIX Dashboard version, if relevant:
apisix-dashboard:2.13-alpine -
Plugin runner version, for issues related to plugin runners:
-
LuaRocks version, for installation issues (run
luarocks --version):/usr/local/bin/luarocks 3.8.0
rewriting the url of grpc request in proxy-rewrite pluing is not supported yet.
And do you set a correct grpc port of the upstream?
rewriting the url of grpc request in proxy-rewrite pluing is not supported yet.
And do you set a correct grpc port of the upstream?
yes, we will, and This error occurs intermittently and is not consistently reproducible.
rewriting the url of grpc request in proxy-rewrite pluing is not supported yet.
And do you set a correct grpc port of the upstream?
@kingluo When will it be supported? Do you have any plan to support it? Also, do you have any recommended ways to proxy gRPC services?
rewriting the url of grpc request in proxy-rewrite pluing is not supported yet. And do you set a correct grpc port of the upstream?
@kingluo When will it be supported? Do you have any plan to support it? Also, do you have any recommended ways to proxy gRPC services?
Yes, we will handle it soon later.
But IMO, changing the URI is not a good practice for grpc requests and seems unnecessary.
The format of grpc URI is /package.service/method, almost all official libraries from main programming langs, e.g. golang, will generate such URI automatically and disallow changing. So it's less possible to generate a non-standard URI like prefixing. So I am wondering why we need to change it.
rewriting the url of grpc request in proxy-rewrite pluing is not supported yet. And do you set a correct grpc port of the upstream?
@kingluo When will it be supported? Do you have any plan to support it? Also, do you have any recommended ways to proxy gRPC services?
Yes, we will handle it soon later.
But IMO, changing the URI is not a good practice for grpc requests and seems unnecessary. The format of grpc URI is
/package.service/method, almost all official libraries from main programming langs, e.g. golang, will generate such URI automatically and disallow changing. So it's less possible to generate a non-standard URI like prefixing. So I am wondering why we need to change it.
I feel like you may have misunderstood my meaning. What we need is to use Apisix to proxy the gRPC service endpoint, for example, our original endpoint is 127.0.0.1:8100 and the http/2 port of Apisix is 127.0.0.1:8002. We want to use 127.0.0.1:8002/xxx/grpc to proxy out the endpoint 127.0.0.1:8100, and it is not necessary to rewrite the gRPC method. The above bug is a problem encountered during the proxy, Apisix often reports the upstream error, it is unstable, sometimes successful and sometimes failed. I guess it may be a bug, can you understand my meaning? @kingluo
@FlyTOmeLight
-
why do you configure proxy-rewrite to change uri? It's confusing and completely unnecessary for grpc.
-
The IP and port are transport-level stuff, which is none of the business of the layer7 protocol, i.e. http2 and grpc based on http2. The downstream IP and port are completely decoupled from the upstream one. So, nothing special to do here, just ensure your upstream is correct.
-
why you configure your upstream as three ports and different protocols?
apiVersion: apisix.apache.org/v2 kind: ApisixUpstream metadata: name: {{ .Release.Name }} spec: loadbalancer: type: ewma retries: {{ .Values.ingress.retries }} timeout: connect: {{ .Values.ingress.timeout.connect }} send: {{ .Values.ingress.timeout.send }} read: {{ .Values.ingress.timeout.read }} portLevelSettings: - port: 8000 scheme: http - port: 8001 scheme: grpc - port: 8002 scheme: http
Anyways, could you show the final configuration via admin API, instead of just template text? So that I could have a full view of your configuration and figure out what's wrong with your configuration.
Here is commands for reference:
# forward admin API port for access
# note that you need to specify the correct namespace you installed apisix
kubectl port-forward --namespace ingress-apisix svc/apisix-admin 9180
# show all routes
curl http://127.0.0.1:9180/apisix/admin/routes -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1'
# show all upstreams
curl http://127.0.0.1:9180/apisix/admin/upstreams -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1'
all routes:
{"node":{"dir":true,"nodes":[{"createdIndex":32,"value":{"uris":["\/*"],"priority":1,"hosts":["dongnan.endpoint.windmill.com"],"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","upstream_id":"b9260e1d","update_time":1681020908,"create_time":1678936960,"name":"default_dongnan_grpc","id":"2c4a379a","labels":{"managed-by":"apisix-ingress-controller"},"status":1},"modifiedIndex":238338,"key":"\/apisix\/routes\/2c4a379a"},{"createdIndex":228312,"value":{"uris":["\/*"],"priority":1,"hosts":["ep-mzinkvxm.endpoint.windmill.com"],"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","upstream_id":"c54210dc","update_time":1681020909,"create_time":1680855294,"name":"default_ep-mzinkvxm_grpc","id":"502e295b","labels":{"managed-by":"apisix-ingress-controller"},"status":1},"modifiedIndex":238347,"key":"\/apisix\/routes\/502e295b"},{"createdIndex":228313,"value":{"uris":["\/ep-mzinkvxm\/metrics\/*"],"priority":2,"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","upstream_id":"5c4b4166","update_time":1681020909,"create_time":1680855294,"name":"default_ep-mzinkvxm_metrics","id":"6064992d","plugins":{"proxy-rewrite":{"use_real_request_uri_unsafe":false,"regex_uri":["^\/ep-mzinkvxm\/metrics\/(.*)","\/$1"]}},"labels":{"managed-by":"apisix-ingress-controller"},"status":1},"modifiedIndex":238348,"key":"\/apisix\/routes\/6064992d"},{"createdIndex":33,"value":{"uris":["\/dongnan\/metrics\/*"],"priority":2,"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","upstream_id":"202f5fa7","update_time":1681020908,"create_time":1678936960,"name":"default_dongnan_metrics","id":"6e97ffb8","plugins":{"proxy-rewrite":{"use_real_request_uri_unsafe":false,"regex_uri":["^\/dongnan\/metrics\/(.*)","\/$1"]}},"labels":{"managed-by":"apisix-ingress-controller"},"status":1},"modifiedIndex":238340,"key":"\/apisix\/routes\/6e97ffb8"},{"createdIndex":31,"value":{"uris":["\/dongnan\/http\/*"],"priority":3,"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","upstream_id":"ce213e8b","update_time":1681020908,"create_time":1678936960,"name":"default_dongnan_http","id":"9076dfa4","plugins":{"api-breaker":{"break_response_code":502,"max_breaker_sec":300,"healthy":{"http_statuses":[200],"successes":1},"unhealthy":{"failure":3,"http_statuses":[500,503],"failures":3}},"proxy-rewrite":{"use_real_request_uri_unsafe":false,"regex_uri":["^\/dongnan\/http\/(.*)","\/$1"]}},"labels":{"managed-by":"apisix-ingress-controller"},"status":1},"modifiedIndex":238337,"key":"\/apisix\/routes\/9076dfa4"},{"createdIndex":228311,"value":{"uris":["\/ep-mzinkvxm\/http\/*"],"priority":3,"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","upstream_id":"b245204a","update_time":1681020909,"create_time":1680855294,"name":"default_ep-mzinkvxm_http","id":"ec12c165","plugins":{"api-breaker":{"break_response_code":502,"max_breaker_sec":300,"healthy":{"http_statuses":[200],"successes":1},"unhealthy":{"failure":3,"http_statuses":[500,503],"failures":3}},"proxy-rewrite":{"use_real_request_uri_unsafe":false,"regex_uri":["^\/ep-mzinkvxm\/http\/(.*)","\/$1"]}},"labels":{"managed-by":"apisix-ingress-controller"},"status":1},"modifiedIndex":238346,"key":"\/apisix\/routes\/ec12c165"}],"key":"\/apisix\/routes"},"count":6,"action":"get"}
all upstreams:
{"node":{"dir":true,"nodes":[{"createdIndex":68328,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903708,"pass_host":"pass","name":"inference_ep-gdivkpvq_8001","id":"1b82663e","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.156","priority":0,"port":8001}],"scheme":"grpc","create_time":1679387520},"modifiedIndex":161948,"key":"\/apisix\/upstreams\/1b82663e"},{"createdIndex":30,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1681021208,"pass_host":"pass","name":"default_dongnan_8002","id":"202f5fa7","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.129","priority":0,"port":8002}],"scheme":"http","create_time":1678936960},"modifiedIndex":238354,"key":"\/apisix\/upstreams\/202f5fa7"},{"createdIndex":192271,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680836108,"pass_host":"pass","name":"inference_ep-uxmeyitu_8000","id":"2040bfe5","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.148","priority":0,"port":8000}],"scheme":"http","create_time":1680249445},"modifiedIndex":227721,"key":"\/apisix\/upstreams\/2040bfe5"},{"createdIndex":21,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680154318,"pass_host":"pass","name":"default_ep-fuphptti_8002","nodes":[],"labels":{"managed-by":"apisix-ingress-controller"},"id":"2132180e","scheme":"http","create_time":1678936959},"modifiedIndex":189383,"key":"\/apisix\/upstreams\/2132180e"},{"createdIndex":48,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903872,"pass_host":"pass","name":"inference_ep-hamvgnxm_8002","nodes":[],"labels":{"managed-by":"apisix-ingress-controller"},"id":"26af8f93","scheme":"http","create_time":1678936961},"modifiedIndex":161961,"key":"\/apisix\/upstreams\/26af8f93"},{"createdIndex":38,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903708,"pass_host":"pass","name":"inference_ep-disgsbce_8001","id":"38f5585c","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.141","priority":0,"port":8001}],"scheme":"grpc","create_time":1678936960},"modifiedIndex":161942,"key":"\/apisix\/upstreams\/38f5585c"},{"createdIndex":15,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680154328,"pass_host":"pass","name":"default_ep-wyvpempc_8002","nodes":[],"labels":{"managed-by":"apisix-ingress-controller"},"id":"41120bc4","scheme":"http","create_time":1678936959},"modifiedIndex":189391,"key":"\/apisix\/upstreams\/41120bc4"},{"createdIndex":37,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903708,"pass_host":"pass","name":"inference_ep-disgsbce_8000","id":"4ff268ca","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.141","priority":0,"port":8000}],"scheme":"http","create_time":1678936960},"modifiedIndex":161941,"key":"\/apisix\/upstreams\/4ff268ca"},{"createdIndex":192272,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680836108,"pass_host":"pass","name":"inference_ep-uxmeyitu_8001","id":"57478f73","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.148","priority":0,"port":8001}],"scheme":"http","create_time":1680249445},"modifiedIndex":227722,"key":"\/apisix\/upstreams\/57478f73"},{"createdIndex":170503,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1681021208,"pass_host":"pass","name":"default_ep-mzinkvxm_8002","id":"5c4b4166","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.171","priority":0,"port":8002}],"scheme":"http","create_time":1679998256},"modifiedIndex":238363,"key":"\/apisix\/upstreams\/5c4b4166"},{"createdIndex":68327,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903708,"pass_host":"pass","name":"inference_ep-gdivkpvq_8000","id":"6c8556a8","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.156","priority":0,"port":8000}],"scheme":"http","create_time":1679387520},"modifiedIndex":161947,"key":"\/apisix\/upstreams\/6c8556a8"},{"createdIndex":68330,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903830,"pass_host":"pass","name":"inference_ep-gdivkpvq_8002","nodes":[],"labels":{"managed-by":"apisix-ingress-controller"},"id":"828b3784","scheme":"http","create_time":1679387520},"modifiedIndex":161957,"key":"\/apisix\/upstreams\/828b3784"},{"createdIndex":39,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903824,"pass_host":"pass","name":"inference_ep-disgsbce_8002","nodes":[],"labels":{"managed-by":"apisix-ingress-controller"},"id":"a1fc09e6","scheme":"http","create_time":1678936960},"modifiedIndex":161953,"key":"\/apisix\/upstreams\/a1fc09e6"},{"createdIndex":13,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680154208,"pass_host":"pass","name":"default_ep-wyvpempc_8000","id":"af1c6ae8","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.148","priority":0,"port":8000}],"scheme":"http","create_time":1678936959},"modifiedIndex":189361,"key":"\/apisix\/upstreams\/af1c6ae8"},{"createdIndex":170500,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1681021208,"pass_host":"pass","name":"default_ep-mzinkvxm_8000","id":"b245204a","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.171","priority":0,"port":8000}],"scheme":"http","create_time":1679998256},"modifiedIndex":238361,"key":"\/apisix\/upstreams\/b245204a"},{"createdIndex":20,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680154208,"pass_host":"pass","name":"default_ep-fuphptti_8001","id":"b83b49b4","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.158","priority":0,"port":8001}],"scheme":"http","create_time":1678936959},"modifiedIndex":189357,"key":"\/apisix\/upstreams\/b83b49b4"},{"createdIndex":29,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1681021208,"pass_host":"pass","name":"default_dongnan_8001","id":"b9260e1d","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.129","priority":0,"port":8001}],"scheme":"http","create_time":1678936960},"modifiedIndex":238352,"key":"\/apisix\/upstreams\/b9260e1d"},{"createdIndex":47,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903708,"pass_host":"pass","name":"inference_ep-hamvgnxm_8001","id":"bfa6de29","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.120","priority":0,"port":8001}],"scheme":"http","create_time":1678936961},"modifiedIndex":161920,"key":"\/apisix\/upstreams\/bfa6de29"},{"createdIndex":170502,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1681021208,"pass_host":"pass","name":"default_ep-mzinkvxm_8001","id":"c54210dc","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.171","priority":0,"port":8001}],"scheme":"grpc","create_time":1679998256},"modifiedIndex":238362,"key":"\/apisix\/upstreams\/c54210dc"},{"createdIndex":46,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903708,"pass_host":"pass","name":"inference_ep-hamvgnxm_8000","id":"c8a1eebf","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.120","priority":0,"port":8000}],"scheme":"http","create_time":1678936961},"modifiedIndex":161918,"key":"\/apisix\/upstreams\/c8a1eebf"},{"createdIndex":28,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1681021208,"pass_host":"pass","name":"default_dongnan_8000","id":"ce213e8b","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.129","priority":0,"port":8000}],"scheme":"http","create_time":1678936960},"modifiedIndex":238350,"key":"\/apisix\/upstreams\/ce213e8b"},{"createdIndex":192274,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680836403,"pass_host":"pass","name":"inference_ep-uxmeyitu_8002","nodes":[],"labels":{"managed-by":"apisix-ingress-controller"},"id":"ce4edec9","scheme":"http","create_time":1680249445},"modifiedIndex":227731,"key":"\/apisix\/upstreams\/ce4edec9"},{"createdIndex":19,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680154208,"pass_host":"pass","name":"default_ep-fuphptti_8000","id":"cf3c7922","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.158","priority":0,"port":8000}],"scheme":"http","create_time":1678936959},"modifiedIndex":189356,"key":"\/apisix\/upstreams\/cf3c7922"},{"createdIndex":14,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680154208,"pass_host":"pass","name":"default_ep-wyvpempc_8001","id":"d81b5a7e","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.148","priority":0,"port":8001}],"scheme":"http","create_time":1678936959},"modifiedIndex":189363,"key":"\/apisix\/upstreams\/d81b5a7e"}],"key":"\/apisix\/upstreams"},"count":24,"action":"get"}
@kingluo
- why do you configure proxy-rewrite to change uri? It's confusing and completely unnecessary for grpc.
we has lots of same modelserver endpoint need to proxy, they have same port, http:8000,grpc:8001, metrics:8002, so we set it in upstreams like this.
kind: ApisixUpstream
metadata:
name: {{ .Release.Name }}
spec:
loadbalancer:
type: ewma
retries: {{ .Values.ingress.retries }}
timeout:
connect: {{ .Values.ingress.timeout.connect }}
send: {{ .Values.ingress.timeout.send }}
read: {{ .Values.ingress.timeout.read }}
portLevelSettings:
- port: 8000
scheme: http
- port: 8001
scheme: grpc
- port: 8002
scheme: http
For grpc upstream server, the scheme must be grpc.
If your service exposes different ports, choose the correct port for grpc only.
Some of your routes use http port but not grpc port.
For example:
this route should use grpc upstream according to what the name hints.
{
"createdIndex":32,
"value":{
"uris":[
"\/*"
],
"priority":1,
"hosts":[
"dongnan.endpoint.windmill.com"
],
"desc":"Created by apisix-ingress-controller, DO NOT modify it manually",
"upstream_id":"b9260e1d",
"update_time":1681020908,
"create_time":1678936960,
"name":"default_dongnan_grpc",
"id":"2c4a379a",
"labels":{
"managed-by":"apisix-ingress-controller"
},
"status":1
},
"modifiedIndex":238338,
"key":"\/apisix\/routes\/2c4a379a"
},
But actually, it uses http upstream:
{
"createdIndex":29,
"value":{
"type":"roundrobin",
"hash_on":"vars",
"desc":"Created by apisix-ingress-controller, DO NOT modify it manually",
"update_time":1681021208,
"pass_host":"pass",
"name":"default_dongnan_8001",
"id":"b9260e1d",
"labels":{
"managed-by":"apisix-ingress-controller"
},
"nodes":[
{
"weight":100,
"host":"10.233.64.129",
"priority":0,
"port":8001
}
],
"scheme":"http",
"create_time":1678936960
},
"modifiedIndex":238352,
"key":"\/apisix\/upstreams\/b9260e1d"
}
Please check your helm cfg according to ingress doc:
https://apisix.apache.org/zh/docs/ingress-controller/concepts/apisix_upstream/
ingress
but i actually use grpc scheme @kingluo
{{- if .Values.ingress.enabled -}}
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
name: {{ .Release.Name }}
spec:
http:
- name: http
priority: 3
match:
paths:
- /{{ .Release.Name }}/http/*
backends:
- serviceName: {{ .Release.Name }}
servicePort: http
plugins:
- name: proxy-rewrite
enable: true
config:
regex_uri: ["^/{{ .Release.Name }}/http/(.*)", "/$1"]
- name: api-breaker
enable: true
config:
break_response_code: 502
unhealthy:
http_statuses: [500, 503]
failure: 3
healthy:
http_statuses: [200]
successes: 1
- name: grpc
priority: 1
match:
hosts:
- {{ .Release.Name }}.endpoint.windmill.com
paths:
- "/*"
backends:
- serviceName: {{ .Release.Name }}
servicePort: grpc
- name: metrics
priority: 2
match:
paths:
- /{{ .Release.Name }}/metrics/*
backends:
- serviceName: {{ .Release.Name }}
servicePort: metrics
plugins:
- name: proxy-rewrite
enable: true
config:
regex_uri: ["^/{{ .Release.Name }}/metrics/(.*)", "/$1"]
---
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
name: {{ .Release.Name }}
spec:
loadbalancer:
type: ewma
retries: {{ .Values.ingress.retries }}
timeout:
connect: {{ .Values.ingress.timeout.connect }}
send: {{ .Values.ingress.timeout.send }}
read: {{ .Values.ingress.timeout.read }}
portLevelSettings:
- port: 8000
scheme: http
- port: 8001
scheme: grpc
- port: 8002
scheme: http
{{- end }}
You should split the ApisixUpstream into multiple upstream specs, one for grpc.
You should split the ApisixUpstream into multiple upstream specs, one for grpc.
i follow by this example and set the apisixupstream spec, you mean i should split it into two specs? it's there any examples? @kingluo
@FlyTOmeLight Could you confirm the route above used the upstream with the wrong scheme?
@tao12345666333, please help, thanks.
What's your APISIX Ingress version?
You can just use admin API to check the results
What's your APISIX Ingress version?
You can just use admin API to check the results
@tao12345666333
ingressbut i actually use grpc scheme @kingluo
{{- if .Values.ingress.enabled -}} apiVersion: apisix.apache.org/v2 kind: ApisixRoute metadata: name: {{ .Release.Name }} spec: http: - name: http priority: 3 match: paths: - /{{ .Release.Name }}/http/* backends: - serviceName: {{ .Release.Name }} servicePort: http plugins: - name: proxy-rewrite enable: true config: regex_uri: ["^/{{ .Release.Name }}/http/(.*)", "/$1"] - name: api-breaker enable: true config: break_response_code: 502 unhealthy: http_statuses: [500, 503] failure: 3 healthy: http_statuses: [200] successes: 1 - name: grpc priority: 1 match: hosts: - {{ .Release.Name }}.endpoint.windmill.com paths: - "/*" backends: - serviceName: {{ .Release.Name }} servicePort: grpc - name: metrics priority: 2 match: paths: - /{{ .Release.Name }}/metrics/* backends: - serviceName: {{ .Release.Name }} servicePort: metrics plugins: - name: proxy-rewrite enable: true config: regex_uri: ["^/{{ .Release.Name }}/metrics/(.*)", "/$1"] --- apiVersion: apisix.apache.org/v2 kind: ApisixUpstream metadata: name: {{ .Release.Name }} spec: loadbalancer: type: ewma retries: {{ .Values.ingress.retries }} timeout: connect: {{ .Values.ingress.timeout.connect }} send: {{ .Values.ingress.timeout.send }} read: {{ .Values.ingress.timeout.read }} portLevelSettings: - port: 8000 scheme: http - port: 8001 scheme: grpc - port: 8002 scheme: http {{- end }}
yes, this is my whole setting @kingluo @tao12345666333 here is the apisixroute and apisixupstream description.
Name: dongnan
Namespace: default
Labels: <none>
Annotations: <none>
API Version: apisix.apache.org/v2
Kind: ApisixRoute
Metadata:
Creation Timestamp: 2023-02-27T02:50:52Z
Generation: 1
Managed Fields:
API Version: apisix.apache.org/v2
Fields Type: FieldsV1
fieldsV1:
f:spec:
Manager: Go-http-client
Operation: Update
Time: 2023-02-27T02:50:52Z
API Version: apisix.apache.org/v2
Fields Type: FieldsV1
fieldsV1:
f:spec:
f:http:
f:status:
.:
f:conditions:
Manager: apisix-ingress-controller
Operation: Update
Time: 2023-02-27T02:50:52Z
Resource Version: 54330836
UID: e58a7211-4153-45a9-b741-bca635bc2aa3
Spec:
Http:
Backends:
Service Name: dongnan
Service Port: http
Match:
Paths:
/dongnan/http/*
Name: http
Plugins:
Config:
regex_uri:
^/dongnan/http/(.*)
/$1
Enable: true
Name: proxy-rewrite
Config:
break_response_code: 502
Healthy:
http_statuses:
200
Successes: 1
Unhealthy:
Failure: 3
http_statuses:
500
503
Enable: true
Name: api-breaker
Priority: 3
Backends:
Service Name: dongnan
Service Port: grpc
Match:
Hosts:
dongnan.endpoint.windmill.com
Paths:
/*
Name: grpc
Priority: 1
Backends:
Service Name: dongnan
Service Port: metrics
Match:
Paths:
/dongnan/metrics/*
Name: metrics
Plugins:
Config:
regex_uri:
^/dongnan/metrics/(.*)
/$1
Enable: true
Name: proxy-rewrite
Priority: 2
Status:
Conditions:
Message: Sync Successfully
Observed Generation: 1
Reason: ResourcesSynced
Status: True
Type: ResourcesAvailable
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ResourcesSynced 2m58s (x3821 over 13d) ApisixIngress ApisixIngress synced successfully
Name: dongnan
Namespace: default
Labels: <none>
Annotations: <none>
API Version: apisix.apache.org/v2
Kind: ApisixUpstream
Metadata:
Creation Timestamp: 2023-02-27T02:50:52Z
Generation: 1
Managed Fields:
API Version: apisix.apache.org/v2
Fields Type: FieldsV1
fieldsV1:
f:spec:
.:
f:loadbalancer:
.:
f:type:
f:portLevelSettings:
f:retries:
f:timeout:
.:
f:connect:
f:read:
f:send:
Manager: Go-http-client
Operation: Update
Time: 2023-02-27T02:50:52Z
Resource Version: 45712201
UID: c8dc04aa-7978-496e-827e-2cd22e475dbc
Spec:
Loadbalancer:
Type: ewma
Port Level Settings:
Port: 8000
Scheme: http
Port: 8001
Scheme: grpc
Port: 8002
Scheme: http
Retries: 3
Timeout:
Connect: 1s
Read: 5s
Send: 5s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ResourcesSynced 3m30s (x3800 over 13d) ApisixIngress ApisixIngress synced successfully
@FlyTOmeLight Could you confirm this route matched the wrong upstream scheme?
https://github.com/apache/apisix/issues/9240#issuecomment-1501064828
That is, you need to check the final configuration representation via admin API, if it is wrong and unexpected, then obviously the ApisixRoute and ApisixUpstream specs are wrong and you need to fix them.
@FlyTOmeLight Could you confirm this route matched the wrong upstream scheme?
I actually set grpc scheme, i give all the config and description in this issue, could you please tell me what's wrong with my setting? @kingluo
I mean, the name of this route contains "grpc", but it points to "http" upstream, is it expected or not?
If it's wrong, then you need to adjust your CRD, clear?
You could try to use one ApisixUpstream spec for each port instead, although they all point to the same k8s service.
I mean, the name of this route contains "grpc", but it points to "http" upstream, is it expected or not?
If it's wrong, then you need to adjust your CRD, clear?
You could try to use one ApisixUpstream spec for each port instead, although they all point to the same k8s service.
Of course, it's not what we expected. I repeatedly submitted our CRD in this issue and also asked you about the way to separate ApisixUpstream. When we submitted the CRD, the 8001 port was supposed to correspond to the gRPC scheme, but the Apisix Admin API returned HTTP instead. Isn't this an issue with Apisix? In addition, it is clearly stated in the Apisix documentation that arrays are supported, and there is no specific emphasis on the need to separate different schemes or services into different upstreams.
@FlyTOmeLight Try ApisixRoute without ApisixUpstream, specify the port directly.
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
name: foo-bar-route
spec:
http:
- name: foo
match:
hosts:
- foo.com
paths:
- "/foo/*"
backends:
- serviceName: foo
servicePort: 8001
@tao12345666333 any idea about ApisixUpstream for multi-ports service?
I conducted a lot of experiments and found that in apisixupstream, multiple schemes cannot be set. It always takes the scheme of the last element in the "portlevelsetting" array. I'm not sure if this is your design or a bug. According to the documentation, it seems like it should be possible to set it up this way.The setting shown in the following figure in the documentation is not effective. @kingluo @tao12345666333
@FlyTOmeLight does your use case work correctly if you use the normal way to specifiy the scheme for the upstream?
