apisix-ingress-controller icon indicating copy to clipboard operation
apisix-ingress-controller copied to clipboard

Published 20 hours ago • verified publisher icon apache

request help: How to configure ApisixPluginConfig？

Open yin6516008 opened this issue 3 years ago • 28 comments

Issue description

CRD resource configuration files：

apiVersion: apisix.apache.org/v2beta3
kind: ApisixPluginConfig
metadata:
  name: openapi-auth
  namespace: default
spec:
  plugins:
    - name: authz-casbin
      enable: true
      config:
        model_path: "/usr/local/apisix/conf/casbin-model.conf"
        policy: "test, /*,GET"
        username: "user"

apisix-ingress-controller log: 2022-02-08T09:43:51+08:00 [33mwarn[0m ingress/apisix_pluginconfig.go:224 sync ApisixPluginConfig failed, will retry {"object": {"Type":2,"Object":{"Key":"default/openapi-auth","OldObject":{},"GroupVersion":"apisix.apache.org/v2beta3"},"Tombstone":null}, "error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to check the configuration of plugin authz-casbin err: value should match only one schema, but matches none\"}\n\n\n"}

Is the CRD configuration fields is there a problem？

Environment

  • your apisix-ingress-controller version (output of apisix-ingress-controller version --long): 1.4
  • your Kubernetes cluster version (output of kubectl version): 1.20
  • if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a): centos7.6

yin6516008 avatar Feb 08 '22 01:02 yin6516008

@yin6516008 The reason is the config of authz-casbin is not correct, it doesn't match the jsonschema requirements.

@tao12345666333 We don't have strong check for plugin config? I think we need to strengthen it.

tokers avatar Feb 08 '22 09:02 tokers

I create succeeded in using the following way：

apiVersion: apisix.apache.org/v2beta3
kind: ApisixPluginConfig
metadata:
  name: openapi-auth
  namespace: default
spec:
  plugins:
    - name: authz-casbin
      enable: true
      config:
        model: "[request_definition]\nr = sub, obj, act\n[policy_definition]\np = sub, obj, act\n[role_definition]\ng = _, _\n[policy_effect]\ne = some(where (p.eft == allow))\n[matchers]\nm = g(r.sub, p.sub) && globMatch(r.obj,p.obj) && regexMatch(r.act,p.act)"
        policy: "p, test, /*, *\np,test2 /*, GET"
        username: "user"

/ingress-apisix # curl "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/plugin_configs" -H 'X-API-KEY: edxxxxxxxxxxxxxxx25c8f1'
{"count":1,"node":{"key":"\/apisix\/plugin_configs","dir":true,"nodes":[{"key":"\/apisix\/plugin_configs\/6191bd21","value":{"plugins":{"authz-casbin":{"model":"[request_definition]\nr = sub, obj, act\n[policy_definition]\np = sub, obj, act\n[role_definition]\ng = _, _\n[policy_effect]\ne = some(where (p.eft == allow))\n[matchers]\nm = g(r.sub, p.sub) && globMatch(r.obj,p.obj) && regexMatch(r.act,p.act)","policy":"p, test, \/foo, GET\np,test2 \/bar, GET","username":"user"}},"labels":{"managed-by":"apisix-ingress-controller"},"id":"6191bd21","update_time":1644314331,"name":"default_openapi-auth","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","create_time":1644311280},"modifiedIndex":62,"createdIndex":41}]},"action":"get"}

create a consumer

apiVersion: apisix.apache.org/v2beta3
kind: ApisixConsumer
metadata:
  name: test
  namespace: default
spec:
  authParameter:
    keyAuth: 
      value: 
        key: ZTLb3X1qmf5e

/ingress-apisix # curl "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/consumers" -H 'X-API-KEY: edxxxxxxxxxxxxc8f1'
{"count":1,"node":{"key":"\/apisix\/consumers","dir":true,"nodes":[{"key":"\/apisix\/consumers\/default_test","value":{"plugins":{"key-auth":{"key":"ZTLb3X1qmf5e"}},"labels":{"managed-by":"apisix-ingress-controller"},"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1644228859,"username":"default_test","create_time":1644228859},"modifiedIndex":38,"createdIndex":38}]},"action":"get"}

But when I quote this plugin in routing, don't take effect.

apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
  name: echo
  namespace: default
spec:
  http:
  - name: echo
    match:
      hosts:
      - echo.jc-ai.cn
      paths:
      - /*
    plugin_config_name: openapi-auth
    backends:
      - serviceName: echo
        servicePort: 8080

plugin config is not in the route:

curl "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/routes/f2035783" -H 'X-API-KEY: ecccccccccxxcc5c8f1'
{"count":1,"node":{"key":"\/apisix\/routes\/f2035783","value":{"name":"default_echo_echo","id":"f2035783","priority":0,"upstream_id":"e4f10124","labels":{"managed-by":"apisix-ingress-controller"},"create_time":1644313158,"update_time":1644313158,"uris":["\/*"],"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","status":1,"hosts":["echo.jc-ai.cn"]}},"action":"get"}

Don`t take request header also can get to success.

[root@skywalking ~]# curl echo.jc-ai.cn/foo


Hostname: echo-5ddc95c657-4m85q

Pod Information:
        -no pod information available-

Server values:
        server_version=nginx: 1.14.2 - lua: 10015

Request Information:
        client_address=172.30.0.57
        method=GET
        real path=/foo
        query=
        request_version=1.1
        request_scheme=http
        request_uri=http://echo.jc-ai.cn:8080/foo

Request Headers:
        accept=*/*
        host=echo.jc-ai.cn
        user-agent=curl/7.29.0
        x-forwarded-for=172.30.0.1
        x-forwarded-host=echo.jc-ai.cn
        x-forwarded-port=9080
        x-forwarded-proto=http
        x-real-ip=172.30.0.1

Request Body:
        -no body in request-

Why response status code is not 401?

yin6516008 avatar Feb 08 '22 10:02 yin6516008

let me check it.

tao12345666333 avatar Feb 16 '22 06:02 tao12345666333

I'm sorry, I won't be free debugging it recently，to start over again today I debug it, but I met this problem.

apply

apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
  name: echo
  namespace: default
spec:
  http:
  - name: echo
    match:
      hosts:
      - echo.jc-ai.cn
      paths:
      - /*
    plugin_config_name: openapi-auth
    backends:
      - serviceName: echo
        servicePort: 8080

# kubectl describe  apisixroute  echo 
Name:         echo
Namespace:    default
Labels:       <none>
Annotations:  API Version:  apisix.apache.org/v2beta3
Kind:         ApisixRoute
Metadata:
  Creation Timestamp:  2022-03-03T06:47:27Z
  Generation:          1
  Resource Version:    29884427
  UID:                 fa77ba15-5ccf-4c54-b87d-7516f6a120e4
Spec:
  Http:
    Backends:
      Service Name:  echo
      Service Port:  8080
    Match:
      Hosts:
        echo.jc-ai.cn
      Paths:
        /*
    Name:  echo
Status:
  Conditions:
    Message:              Sync Successfully
    Observed Generation:  1
    Reason:               ResourcesSynced
    Status:               True
    Type:                 ResourcesAvailable
Events:
  Type    Reason           Age                From           Message
  ----    ------           ----               ----           -------
  Normal  ResourcesSynced  11m (x2 over 11m)  ApisixIngress  ApisixIngress synced successfully

Why are there no plugin_config_name field？

yin6516008 avatar Mar 03 '22 07:03 yin6516008

@yin6516008 How was your CRD installed?

Can you use this CRD? https://github.com/apache/apisix-helm-chart/blob/master/charts/apisix-ingress-controller/crds/customresourcedefinitions.yaml

tao12345666333 avatar Mar 03 '22 08:03 tao12345666333

image

I use the latest crd file. but apisixroute dispaly v2beta1. I delete v2betav1、v2betav2 in the apisixroute, apply the route agin, the plugin_config_name appeared.

Let's ignore the situation above. to pay attention to the following questions

apiVersion: apisix.apache.org/v2beta3
kind: ApisixPluginConfig
metadata:
  name: openapi-auth
  namespace: default
spec:
  plugins:
    - name: authz-casbin
      enable: true
      config:
        model: "[request_definition]\nr = sub, obj, act\n[policy_definition]\np = sub, obj, act\n[role_definition]\ng = _, _\n[policy_effect]\ne = some(where (p.eft == allow))\n[matchers]\nm = g(r.sub, p.sub) && globMatch(r.obj,p.obj) && regexMatch(r.act,p.act)"
        policy: "p,test,/foo,POST\np,test2,/bar,POST"
        username: "user"

When I apply this file, apisix-ingress-controller got below error

2022-03-04T14:35:14+08:00	info	ingress/controller.go:609	service kube-system/ccm not found
W0304 14:35:15.141661       1 reflector.go:441] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: watch of *v2beta3.ApisixPluginConfig ended with: an error on the server ("unable to decode an event from the watch stream: unable to decode watch event: no kind \"ApisixPluginConfig\" is registered for version \"apisix.apache.org/v2beta3\" in scheme \"pkg/runtime/scheme.go:100\"") has prevented the request from succeeding
[GIN] 2022/03/04 - 14:35:15 | 200 |       34.34µs |      172.30.0.1 | GET      "/healthz"
W0304 14:35:16.287189       1 reflector.go:441] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: watch of *v2beta3.ApisixPluginConfig ended with: an error on the server ("unable to decode an event from the watch stream: unable to decode watch event: no kind \"ApisixPluginConfig\" is registered for version \"apisix.apache.org/v2beta3\" in scheme \"pkg/runtime/scheme.go:100\"") has prevented the request from succeeding
2022-03-04T14:35:16+08:00	info	ingress/controller.go:609	service kube-system/ccm not found

I have tried many ways to no avail. Is there a problem with the format?

yin6516008 avatar Mar 04 '22 06:03 yin6516008

image

I use the latest crd file. but apisixroute dispaly v2beta1. I delete v2betav1、v2betav2 in the apisixroute, apply the route agin, the plugin_config_name appeared.

The CRD version is the cause of the problem.

This feature was only introduced in the latest version.

As for the second question you mentioned, we will use the #873 for tracking

tao12345666333 avatar Mar 04 '22 10:03 tao12345666333

In previous versions, it can be configured as follows


apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
 name: httpbin-route
spec:
 http:
 - name: rule1
   match:
     hosts:
     - httpbin.org
     paths:
       - /ip
   backends:
   - serviceName: foo
     servicePort: 9090
     weight: 10
   plugins:
   - name: xxxx
     enable: true
     config:
       Xxxx

tao12345666333 avatar Mar 04 '22 10:03 tao12345666333

plugins do not meet my needs. I wait for the result of #873

image I use https://github.com/apache/apisix-helm-chart/blob/master/charts/apisix-ingress-controller/crds/customresourcedefinitions.yaml there are three versions of aipsixroute, they are v2beta1、v2beta2、v2beta3。 image I had to delete v2beta1 and v2beta2, apply the yaml file again to see the plugin_config_name

yin6516008 avatar Mar 04 '22 10:03 yin6516008

OK. Other versions are kept for compatibility.

tao12345666333 avatar Mar 04 '22 11:03 tao12345666333

@yin6516008 @tao12345666333 I have the same problem. Could you tell me how to delete the v2beta1 and v2beta2?

xianshun163 avatar Mar 22 '22 10:03 xianshun163

You can modify the CRD configuration to remove unwanted versions

tao12345666333 avatar Mar 22 '22 10:03 tao12345666333

@tao12345666333 use the ApisixPluginConfig , make the apisix ingress controller panic:

W0322 19:03:41.005468 1 reflector.go:441] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: watch of *v2beta3.ApisixPluginConfig ended with: an error on the server ("unable to decode an event from the watch stream: unable to decode watch event: no kind "ApisixPluginConfig" is registered for version "apisix.apache.org/v2beta3" in scheme "pkg/runtime/scheme.go:100"") has prevented the request from succeeding W0322 19:03:41.012135 1 reflector.go:441] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: watch of *v2beta3.ApisixUpstream ended with: an error on the server ("unable to decode an event from the watch stream: unable to decode watch event: no kind "ApisixUpstream" is registered for version "apisix.apache.org/v2beta3" in scheme "pkg/runtime/scheme.go:100"") has prevented the request from succeeding E0322 19:03:41.904002 1 runtime.go:76] Observed a panic: invalid ApisixPluginConfig type goroutine 553 [running]: k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1a0bbc0, 0x1f7f950) /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0x95 k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x86 panic(0x1a0bbc0, 0x1f7f950) /usr/local/go/src/runtime/panic.go:965 +0x1b9 github.com/apache/apisix-ingress-controller/pkg/kube.MustNewApisixPluginConfig(...) /build/controller/pkg/kube/apisix_plugin_config.go:109 github.com/apache/apisix-ingress-controller/pkg/ingress.(*apisixPluginConfigController).onDelete(0xc0006fc560, 0x1b83860, 0xc002d0a960) /build/controller/pkg/ingress/apisix_pluginconfig.go:306 +0x719 k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...) /go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:245 k8s.io/client-go/tools/cache.(*processorListener).run.func1() /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:779 +0x166 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00034bf60) /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000aaaf60, 0x1f9e2a0, 0xc0007882d0, 0x1a8e201, 0xc0004e08a0) /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0x9b k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00034bf60, 0x3b9aca00, 0x0, 0x1, 0xc0004e08a0) /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98 k8s.io/apimachinery/pkg/util/wait.Until(...) /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 k8s.io/client-go/tools/cache.(*processorListener).run(0xc000132a00) /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:771 +0x95 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0003c3330, 0xc000418450) /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x51 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x65 panic: invalid ApisixPluginConfig type [recovered] panic: invalid ApisixPluginConfig type

goroutine 553 [running]: k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x109 panic(0x1a0bbc0, 0x1f7f950) /usr/local/go/src/runtime/panic.go:965 +0x1b9 github.com/apache/apisix-ingress-controller/pkg/kube.MustNewApisixPluginConfig(...) /build/controller/pkg/kube/apisix_plugin_config.go:109 github.com/apache/apisix-ingress-controller/pkg/ingress.(*apisixPluginConfigController).onDelete(0xc0006fc560, 0x1b83860, 0xc002d0a960) /build/controller/pkg/ingress/apisix_pluginconfig.go:306 +0x719 k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...) /go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:245 k8s.io/client-go/tools/cache.(*processorListener).run.func1() /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:779 +0x166 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00034bf60) /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002f0bf60, 0x1f9e2a0, 0xc0007882d0, 0x1a8e201, 0xc0004e08a0) /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0x9b k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00034bf60, 0x3b9aca00, 0x0, 0x1, 0xc0004e08a0) /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98 k8s.io/apimachinery/pkg/util/wait.Until(...) /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 k8s.io/client-go/tools/cache.(*processorListener).run(0xc000132a00) /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:771 +0x95 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0003c3330, 0xc000418450) /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x51 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x65

xianshun163 avatar Mar 22 '22 11:03 xianshun163

@xianshun163 what's your APISIX Ingress version? In the current master branch, we have fixed some issues related to ApisixPluginConfig

tao12345666333 avatar Mar 22 '22 12:03 tao12345666333

my apisix ingress version is 1.14

xianshun163 avatar Mar 23 '22 02:03 xianshun163

@xianshun163 Can you give me a minimal steps to reproduce?

tao12345666333 avatar Mar 24 '22 00:03 tao12345666333

I can not do it again just now because the env is used by others. I wil goto deploy another k8s env to reproduce it later.

xianshun163 avatar Mar 24 '22 09:03 xianshun163

thanks

tao12345666333 avatar Mar 24 '22 11:03 tao12345666333

@xianshun163 Can you give me a minimal steps to reproduce? reproduce:

@tao12345666333 my env k8s：v1.19.3 apisix： 2.11.1 apisix ingress controller： helm install apisix-ingress-controller apisix/apisix-ingress-controller --version=0.9.0 --namespace ingress-apisix --set config.kubernetes.ingressVersion=networking/v1beta1
--set config.apisix.serviceName=apisix-gw-lb --set config.apisix.serviceNamespace=default --set config.apisix.servicePort=9080 --set config.logLevel=warn --set config.apisix.adminKey=edd1c9f034335f136f87ad84b625c8f1

I reproduce it like this : 1、kubectl apply -f xsapp-route-pluginconfig.yaml

2、kubectl apply -f customresourcedefinitions.yaml.bak （I delete the old version of apisixroute）

3、kubectl delete -f xsapp-route-pluginconfig.yaml

then the ingress controller will panic ,and restart .

xsapp-route-pluginconfig.zip

customresourcedefinitions.yaml.zip

xianshun163 avatar Mar 25 '22 08:03 xianshun163

thanks， let me check

tao12345666333 avatar Mar 26 '22 00:03 tao12345666333

@tao12345666333 I have know why the pluginconfig is not effect in the route,

becarse the apisix ingress controller who was deploy by helm use the v2beta1 default. I redeploy it and add the parameter: --set config.kubernetes.apisixRouteVersion=apisix.apache.org/v2beta3 is ok now.

the route is like below, it have a plugin_config_id , and it work well.

{ "uris": [ "/arch/xsapp/*" ], "name": "default_xsapp-route_rule1", "desc": "Created by apisix-ingress-controller, DO NOT modify it manually", "hosts": [ "local.xsapp.org" ], "plugin_config_id": "4682152b", "upstream_id": "db067616", "labels": { "managed-by": "apisix-ingress-controller" }, "status": 1 }

xianshun163 avatar Mar 29 '22 09:03 xianshun163

@xianshun163 thanks for your feedback! 👍

tao12345666333 avatar Mar 29 '22 09:03 tao12345666333

@tao12345666333 After I updated the apisix ingress controller according to the above steps, a seemingly serious problem occurred。 All my routes have been added a configuration: plugin_config_id. I only create ApisixUpstream and ApisixRoute , but the the apisix ingress controller create the plugin config auto. Sometimes, I create tow routes for a Microservices( as a k8s svc). every routes have its own plugin . But the plugin config who auto create is only one name （k8s svc name) . They will cover each other.

kubectl get ApisixPluginConfig -A No resources found

but the dashboard can see all the plugin config: image

my yaml file like this:

apiVersion: apisix.apache.org/v2beta3 kind: ApisixUpstream metadata: name: xsapp-svc namespace: test-xs spec: timeout: connect: 10s read: 60s send: 60s

apiVersion: apisix.apache.org/v2beta3 kind: ApisixRoute metadata: name: xs-privacy namespace: test-xs spec: http: - name: rule1 match: hosts: - api.dev.xs.cn - api.dev.xs.com paths: - /xs/xs-privacy/* backends: - serviceName: xsapp-svc servicePort: 80 plugins: - name: proxy-rewrite enable: true config: regex_uri: - "^/xs/xs-privacy/*" - "/xs/test.svc/${1}"

xianshun163 avatar Mar 29 '22 11:03 xianshun163

@xianshun163 This is a bug https://github.com/apache/apisix-ingress-controller/issues/836 and has been fixed in the latest code. #845

tao12345666333 avatar Mar 30 '22 01:03 tao12345666333

@tao12345666333 I see that version 1.5 will not be released for some time. Can you tell me where the dockerfile of the official package image is? I want to do it myself. Do you think it's better to package with the master branch or add the bug repair code based on version 1.4? If the crd has change in v1.5, I use the v2beta3 version now.

thanks.

xianshun163 avatar Mar 30 '22 03:03 xianshun163

@xianshun163 you can just run

make build-image

To build your docker image.

We can discuss whether we need to release 1.4.1 at our community meeting this afternoon

tao12345666333 avatar Mar 30 '22 04:03 tao12345666333

ok thanks

xianshun163 avatar Mar 30 '22 04:03 xianshun163

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

github-actions[bot] avatar Jul 31 '22 01:07 github-actions[bot]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.

github-actions[bot] avatar Aug 31 '22 01:08 github-actions[bot]