request help: apisix-ingress-controller "error": "unexpected non-empty object"

[renweijun]

Issue description

apisix集群物理机部署非容器环境，这套集群是否可以同时作为apisix-ingress-controller 的apisix/admin 使用，又提供给外部物理机环境通过Dashboard配置路由，上游。既我想容器和物理机混用apisix集群。 实测下来好像不可以，不知道是否支持？有没有计划支持？ apisix-ingress-controller 报错信息 2021-10-13T16:15:38+08:00 error apisix/upstream.go:121 failed to convert upstream item {"url": "http://192.168.8.9:9080/apisix/admin/upstreams", "upstream_key": "/apisix/upstreams/376788208932881080", "error": "unexpected non-empty object"} 2021/10/13 下午4:15:38 2021-10-13T16:15:38+08:00 error apisix/cluster.go:199 failed to list upstreams in APISIX: unexpected non-empty object 2021/10/13 下午4:15:40 2021-10-13T16:15:40+08:00 error apisix/upstream.go:121 failed to convert upstream item {"url": "http://192.168.8.9:9080/apisix/admin/upstreams", "upstream_key": "/apisix/upstreams/376788208932881080", "error": "unexpected non-empty object"} 2021/10/13 下午4:15:40 2021-10-13T16:15:40+08:00 error apisix/cluster.go:199 failed to list upstreams in APISIX: unexpected non-empty object 2021/10/13 下午4:15:42 [GIN] 2021/10/13 - 16:15:42 | 200 | 34.554µs | ip | GET "/healthz" 2021/10/13 下午4:15:42 2021-10-13T16:15:42+08:00 error apisix/upstream.go:121 failed to convert upstream item {"url": "http://192.168.8.9:9080/apisix/admin/upstreams", "upstream_key": "/apisix/upstreams/376788208932881080", "error": "unexpected non-empty object"} 2021/10/13 下午4:15:42 2021-10-13T16:15:42+08:00 error apisix/cluster.go:199 failed to list upstreams in APISIX: unexpected non-empty object 2021/10/13 下午4:15:44 2021-10-13T16:15:44+08:00 error apisix/upstream.go:121 failed to convert upstream item {"url": "http://192.168.8.9:9080/apisix/admin/upstreams", "upstream_key": "/apisix/upstreams/376788208932881080", "error": "unexpected non-empty object"} 2021/10/13 下午4:15:44 2021-10-13T16:15:44+08:00 error apisix/cluster.go:199 failed to list upstreams in APISIX: unexpected non-empty object 2021/10/13 下午4:15:44 [GIN] 2021/10/13 - 16:15:44 | 200 | 37.139µs | ip | GET "/healthz" 2021/10/13 下午4:15:46 2021-10-13T16:15:46+08:00 error apisix/upstream.go:121 failed to convert upstream item {"url": "http://192.168.8.9:9080/apisix/admin/upstreams", "upstream_key": "/apisix/upstreams/376788208932881080", "error": "unexpected non-empty object"} 2021/10/13 下午4:15:46 2021-10-13T16:15:46+08:00 error apisix/cluster.go:199 failed to list upstreams in APISIX: unexpected non-empty object 2021/10/13 下午4:15:46 2021-10-13T16:15:46+08:00 error apisix/cluster.go:161 failed to sync cache {"cost_time": "8.039422647s", "cluster": "default"} 2021/10/13 下午4:15:46 2021-10-13T16:15:46+08:00 error ingress/controller.go:391 failed to wait the default cluster to be ready: unexpected non-empty object 2021/10/13 下午4:15:46 2021-10-13T16:15:46+08:00 info ingress/controller.go:332 controller now is running as a candidate {"namespace": "ingress-apisix", "pod": "apisix-ingress-controller-64f9645f9c-dfp9p"} 2021/10/13 下午4:15:46 I1013 16:15:46.528768 1 leaderelection.go:243] attempting to acquire leader lease ingress-apisix/ingress-apisix-leader... 2021/10/13 下午4:15:46 2021-10-13T16:15:46+08:00 info apisix/cluster.go:152 syncing cache {"cluster": "default"} 2021/10/13 下午4:15:46 2021-10-13T16:15:46+08:00 info apisix/cluster.go:340 syncing schema {"cluster": "default"} 2021/10/13 下午4:15:46 2021-10-13T16:15:46+08:00 error apisix/plugin.go:46 failed to list plugins' names: Get http://192.168.8.9:9080/apisix/admin/plugins/list: context canceled 2021/10/13 下午4:15:46 2021-10-13T16:15:46+08:00 error apisix/cluster.go:360 failed to list plugin names in APISIX: Get http://192.168.8.9:9080/apisix/admin/plugins/list: context canceled 2021/10/13 下午4:15:46 2021-10-13T16:15:46+08:00 warn apisix/cluster.go:324 failed to sync schema: Get http://192.168.8.9:9080/apisix/admin/plugins/list: context canceled 2021/10/13 下午4:15:46 2021-10-13T16:15:46+08:00 error apisix/route.go:117 failed to list routes: Get http://192.168.8.9:9080/apisix/admin/routes: context canceled 2021/10/13 下午4:15:46 2021-10-13T16:15:46+08:00 error apisix/cluster.go:194 failed to list route in APISIX: Get http://192.168.8.9:9080/apisix/admin/routes: context canceled

其中/apisix/upstreams/376788208932881080 是我在dashboard上手动创建的上游 ，dashboard 删除后，ingress可以正常自动注册

Environment

apisix 2.8 dashboard 2.8
外部物理机部署的apisix

• your apisix-ingress-controller version (output of apisix-ingress-controller version --long); Version: 1.2.0 Git SHA: no-git-module Go Version: go1.13.8 Building OS/Arch: linux/amd64 Running OS/Arch: linux/amd64
• your Kubernetes cluster version (output of kubectl version); Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.20", GitCommit:"1f3e19b7beb1cc0110255668c4238ed63dadb7ad", GitTreeState:"clean", BuildDate:"2021-06-16T12:58:51Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
• if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a). 5.4.151-1.el7.elrepo.x86_64 #1 SMP Tue Oct 5 10:21:01 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

[gxthrj]

It is theoretically feasible to let APISIX serve the k8s cluster and other virtual machine clusters at the same time. But there are a few things to note:

1. The ingress-controller needs to be able to access the APISIX admin API normally, and you need to ensure that the network is smooth. Of cause, APISIX also needs to be able to access k8s internal services
2. The dashboard only manages the virtual machine cluster, not the k8s cluster. The configuration management in the k8s cluster is handled by the ingress controller.

[renweijun]

I know what you said, I'm in the container curl APISIX admin API is OK 。but now I need to solve the error report. Thank you

@gxthrj

[gxthrj]

Please show the response of the request to APISIX admin API from the ingress-controller container.

[renweijun]

@gxthrj

[gxthrj]

Judging from the log, it should be that the upstream created through the dashboard cannot be converted to the upstream data structure in the ingress-controller.

@renweijun I will try to simulate your scenario.

[renweijun]

I hope it can be settled as soon as possible，thanks @gxthrj

[gxthrj]

The routing object id created by the dashboard is of numeric type, but the id in ingress is of string type, so some changes are needed to prevent ingress from reporting errors.

[tao12345666333]

There may be some historical data compatibility issues, we may need some discussion. I will put it in the v1.5 plan.

[gxthrj]

This is still a bit troublesome. We have to figure out the scope of historical data. We may not be compatible with the old version. I suggest that we can distinguish it according to the LTS version. We only support the LTS version and the latest version of APISIX.

[namdiag]

When i set image tag to 1.4.0 i get the error apisix/upstream.go:122 failed to convert upstream item {"url": "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams", "upstream_key": "/apisix/upstreams/f7a2533e", "error": "unexpected non-empty object"} but it's not legacy upstream objects either, it's the upstream object create by using ApisixRoute

[tao12345666333]

@namdiag Did you upgrade the CRD configuration?

[namdiag]

@tao12345666333 No i'm not update anything. After apply an ApisixRoute object, i check dashboard and didn't see upstream object, check log and it has same error as in the topic question. After check comment i set image of ingress controller to 1.4.0 tag, then this error appear

[tao12345666333]

@namdiag Please upgrade CRD if you want to use v1.4. We have received some similar feedback for issues caused by not upgrade the CRD.

[renweijun]

@gxthrj What is the version 1.5 update plan？

[tao12345666333]

@renweijun plz see #978

[mhh12121]

Almost same issue here, I manually installed apisix2.13.1-alpine , apisix-admin2.11-alpine through apisix-helm-chart repo.

After ingress-controller-1.4.1 installed, I found the following logs:

2022-05-16T15:38:35+08:00	info	ingress/ingress.go:120	use configuration
{
	"cert_file": "/etc/webhook/certs/cert.pem",
	"key_file": "/etc/webhook/certs/key.pem",
	"log_level": "info",
	"log_output": "stderr",
	"http_listen": ":8080",
	"https_listen": ":8443",
	"ingress_publish_service": "",
	"ingress_status_address": [],
	"enable_profiling": true,
	"kubernetes": {
		"kubeconfig": "",
		"resync_interval": "6h0m0s",
		"app_namespaces": [
			""
		],
		"namespace_selector": [],
		"election_id": "ingress-apisix-leader",
		"ingress_class": "apisix",
		"ingress_version": "networking/v1",
		"watch_endpoint_slices": false,
		"apisix_route_version": "apisix.apache.org/v2beta1",
		"enable_gateway_api": false
	},
	"apisix": {
		"default_cluster_name": "default",
		"default_cluster_base_url": "http://apisix-admin.default.svc.cluster.local:9180/apisix/admin",
		"default_cluster_admin_key": "edd1c9f034335f136f87ad84b625c8f1",
		"base_url": "",
		"admin_key": ""
	}
}
2022-05-16T15:38:35+08:00	warn	api/server.go:75	failed to load x509 key pair, will not start admission server	{"Error": "open /etc/webhook/certs/cert.pem: no such file or directory", "CertFilePath": "/etc/webhook/certs/cert.pem", "KeyFilePath": "/etc/webhook/certs/key.pem"}
I0516 15:38:35.316897       1 leaderelection.go:248] attempting to acquire leader lease default/ingress-apisix-leader...
2022-05-16T15:38:35+08:00	warn	ingress/controller.go:358	found a new leader apisix-ingress-controller-5ff64f78cc-q9922
2022-05-16T15:38:35+08:00	info	ingress/controller.go:360	controller now is running as a candidate	{"namespace": "default", "pod": "apisix-ingress-controller-f548f9988-9h7ms"}
[GIN] 2022/05/16 - 15:38:41 | 200 |      68.296µs |     10.23.4.247 | GET      "/healthz"
[GIN] 2022/05/16 - 15:38:41 | 200 |      48.229µs |     10.23.4.247 | GET      "/healthz"
[GIN] 2022/05/16 - 15:38:51 | 200 |      34.403µs |     10.23.4.247 | GET      "/healthz"
[GIN] 2022/05/16 - 15:38:51 | 200 |      35.595µs |     10.23.4.247 | GET      "/healthz"
2022-05-16T15:38:52+08:00	info	ingress/controller.go:320	LeaderElection	{"message": "apisix-ingress-controller-f548f9988-9h7ms became leader", "event_type": "Normal"}
I0516 15:38:52.715306       1 leaderelection.go:258] successfully acquired lease default/ingress-apisix-leader
2022-05-16T15:38:52+08:00	warn	ingress/controller.go:358	found a new leader apisix-ingress-controller-f548f9988-9h7ms
2022-05-16T15:38:52+08:00	info	ingress/controller.go:409	controller tries to leading ...	{"namespace": "default", "pod": "apisix-ingress-controller-f548f9988-9h7ms"}
2022-05-16T15:38:52+08:00	info	apisix/cluster.go:158	syncing cache	{"cluster": "default"}
2022-05-16T15:38:52+08:00	info	apisix/cluster.go:370	syncing schema	{"cluster": "default"}
2022-05-16T15:38:52+08:00	warn	apisix/cluster.go:330	waiting cluster default to ready, it may takes a while
2022-05-16T15:38:52+08:00	error	apisix/upstream.go:122	failed to convert upstream item	{"url": "http://apisix-admin.default.svc.cluster.local:9180/apisix/admin/upstreams", "upstream_key": "/apisix/upstreams/406931970073822084", "error": "unexpected non-empty object"}

[tokers]

Do you have some upstream objects created by APISIX Admin API or Manager API?

[mhh12121]

Yes, before apply the apisix-ingress, I've already put some upstream through adminUI;

[tokers]

I think this is due to the upstream node format (object vs array). We don't support the object format for the node.

[mhh12121]

So we can say the ingress now is incompatible with admin dashboard? We could only choose one between them ? Before I turn to ingress, I should remove all of the upstreams created from dashboard ?

[tao12345666333]

So we can say the ingress now is incompatible with admin dashboard?

We recommend that Dashboard only be used as a viewing tool when using APISIX Ingress

[tao12345666333]

There may be some historical data compatibility issues, we may need some discussion. I will put it in the v1.5 plan.

#1222 has fixed the data compatibility issues, and I create a PR #1250 to add cherry-pick it to v1.5

[tao12345666333]

I think the original problem has been fixed, I will close this one. Thanks all.