kcp icon indicating copy to clipboard operation
kcp copied to clipboard

Published 20 hours ago • verified publisher icon kcp-dev

flake: TestAPIExportAPIBindingsAccess

Open stevekuznetsov opened this issue 3 years ago • 15 comments 

--- FAIL: TestAPIExportAPIBindingsAccess (51.30s)
    kcp.go:130: shared kcp server will target configuration "/home/runner/work/kcp/kcp/.kcp/admin.kubeconfig"
    workspaces.go:191: Created organization workspace root:e2e-org-p2l7n
    workspaces.go:124: Created root:universal workspace root:e2e-org-p2l7n:workspace1
    workspaces.go:124: Created root:universal workspace root:e2e-org-p2l7n:workspace2
    virtualworkspace_test.go:359: root:e2e-org-p2l7n:workspace1: creating APIResourceSchema 1
    virtualworkspace_test.go:360: root:e2e-org-p2l7n:workspace1: creating APIExport 1
    virtualworkspace_test.go:361: root:e2e-org-p2l7n:workspace1: creating APIResourceSchema 2
    virtualworkspace_test.go:362: root:e2e-org-p2l7n:workspace1: creating APIExport 2
    virtualworkspace_test.go:363: root:e2e-org-p2l7n:workspace1: creating APIBinding referencing APIExport 1
    virtualworkspace_test.go:364: root:e2e-org-p2l7n:workspace1: creating APIBinding referencing APIExport 2
    virtualworkspace_test.go:366: root:e2e-org-p2l7n:workspace2: creating APIBinding referencing APIExport 1
    virtualworkspace_test.go:375: root:e2e-org-p2l7n:workspace2: creating APIResourceSchema 3
    virtualworkspace_test.go:376: root:e2e-org-p2l7n:workspace2: creating APIExport 3
    virtualworkspace_test.go:377: root:e2e-org-p2l7n:workspace2: creating APIBinding referencing APIExport 3
    virtualworkspace_test.go:421: Verifying APIExport root:e2e-org-p2l7n:workspace1|export1 discovery has apis.kcp.dev
    virtualworkspace_test.go:421: Verifying APIExport export1 discovery has apibindings
    virtualworkspace_test.go:422: Verifying APIExport root:e2e-org-p2l7n:workspace1|export2 discovery has apis.kcp.dev
    virtualworkspace_test.go:422: Verifying APIExport export2 discovery has apibindings
    virtualworkspace_test.go:423: Verifying APIExport root:e2e-org-p2l7n:workspace2|export3 discovery has apis.kcp.dev
    virtualworkspace_test.go:423: Verifying APIExport export3 discovery has apibindings
    virtualworkspace_test.go:446: Verifying APIExport 1 only serves APIBinding 1|1 and 2|1
    virtualworkspace_test.go:463: Verifying APIExport 2 only serves APIBinding 2
    virtualworkspace_test.go:477: Updating APIExport 1 to claim APIBindings
    virtualworkspace_test.go:491: Updating APIBinding 1 to accept APIBindings claim
    virtualworkspace_test.go:516: Verifying APIExport 1 serves APIBindings 1|1, 1|2, and 2|1
    virtualworkspace_test.go:542: Verifying APIExport 2 still only serves APIBinding 2
    virtualworkspace_test.go:556: Updating APIBinding 1 to reject APIBindings claim
    virtualworkspace_test.go:573: Verifying APIExport 1 back to only serving its own bindings
    assertions.go:1691: Waiting for condition, but got: unexpected binding root:e2e-org-p2l7n:workspace1|binding2
    virtualworkspace_test.go:574: 
        	Error Trace:	util.go:298
        	            				virtualworkspace_test.go:434
        	            				virtualworkspace_test.go:574
        	Error:      	Condition never satisfied
        	Test:       	TestAPIExportAPIBindingsAccess
        	Messages:   	did not see expected bindings for root:e2e-org-p2l7n:workspace1|export1

/cc @ncdc

stevekuznetsov avatar Oct 25 '22 17:10 stevekuznetsov

Logs for any workspace under the org, for reconcilers:

kcp-apibinder-initializer
kcp-apibinding
kcp-apibindingdeletion
kcp-apiexport
kcp-permissionclaimlabel
kcp-resource-permissionclaimlabel
kcp-virtual-apiexport-api-reconciler

kcp.log

stevekuznetsov avatar Oct 26 '22 12:10 stevekuznetsov

Logs for controllers interacting with the APIBinding and APIExport in root:e2e-org-p2l7n:workspace1:

kcp.log 
09:59:48.366700   30606 apiexport_apireconciler_controller.go:172] "queueing APIExport" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:48.366853   30606 apiexport_apireconciler_controller.go:220] "processing key" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:48.366921   30606 apiexport_apireconciler_reconcile.go:53] "no APIs found for API domain key" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1" apiDomainKey=root:e2e-org-p2l7n:workspace1/export1 apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion=""
09:59:48.411439   30606 apiexport_controller.go:212] "queueing APIExport" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:48.411479   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:48.626905   30606 apiexport_apireconciler_controller.go:172] "queueing APIExport" reconciler="kcp-virtual-apiexport-api-reconciler" reason="APIResourceSchema change" apiResourceSchema="v1.widgets.anotherexample.io" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:48.626961   30606 apiexport_apireconciler_controller.go:220] "processing key" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:48.627025   30606 apiexport_apireconciler_reconcile.go:53] "no APIs found for API domain key" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1" apiDomainKey=root:e2e-org-p2l7n:workspace1/export1 apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion=""
09:59:48.914737   30606 apiexport_reconcile.go:149] "creating identity secret" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" secret.workspace="root:e2e-org-p2l7n:workspace1" secret.namespace="kcp-system" secret.name="export1" secret.apiVersion=""
09:59:49.187798   30606 committer.go:113] "patching *v1alpha1.APIExport" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" patch="{\"metadata\":{\"resourceVersion\":\"160111\",\"uid\":\"8213c2cf-940f-4dd7-ae3f-3cad407cf27b\"},\"spec\":{\"identity\":{\"secretRef\":{\"name\":\"export1\",\"namespace\":\"kcp-system\"}}}}"
09:59:49.195394   30606 apibinding_controller.go:260] "queueing APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:49.195407   30606 permissionclaimlabel_controller.go:118] "queueing APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:49.201124   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:49.202401   30606 permissionclaimlabel_reconcile.go:90] "claim set details" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" expected=map[] unexpected=map[] toApply=map[] toRemove=map[] all=map[]
09:59:49.202596   30606 permissionclaimlabel_controller.go:221] "patching APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" patch="{\"metadata\":{\"resourceVersion\":\"160130\",\"uid\":\"6a3dddf3-5b8a-4563-b1db-5c149dd936bd\"},\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2022-10-25T09:59:49Z\",\"status\":\"True\",\"type\":\"PermissionClaimsApplied\"},{\"lastTransitionTime\":\"2022-10-25T09:59:49Z\",\"status\":\"True\",\"type\":\"PermissionClaimsValid\"}]}}"
09:59:49.329992   30606 apiexport_apireconciler_controller.go:172] "queueing APIExport" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:49.330050   30606 apiexport_apireconciler_controller.go:220] "processing key" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:49.330091   30606 apiexport_apireconciler_reconcile.go:53] "no APIs found for API domain key" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1" apiDomainKey=root:e2e-org-p2l7n:workspace1/export1 apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion=""
09:59:49.332428   30606 apibinding_controller.go:260] "queueing APIBinding because of APIExport" reconciler="kcp-apibinding" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:49.332492   30606 apiexport_controller.go:212] "queueing APIExport" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:49.332515   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:49.332764   30606 committer.go:113] "patching *v1alpha1.APIExport" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" patch="{\"metadata\":{\"resourceVersion\":\"160136\",\"uid\":\"8213c2cf-940f-4dd7-ae3f-3cad407cf27b\"},\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2022-10-25T09:59:49Z\",\"status\":\"True\",\"type\":\"IdentityValid\"},{\"lastTransitionTime\":\"2022-10-25T09:59:49Z\",\"status\":\"True\",\"type\":\"VirtualWorkspaceURLsReady\"}],\"identityHash\":\"a81718b4324e2c8626859fef2d6e18159cb13117bc861df955b09310d0468335\",\"virtualWorkspaces\":[{\"url\":\"https://10.1.0.57:6444/services/apiexport/root:e2e-org-p2l7n:workspace1/export1\"},{\"url\":\"https://10.1.0.57:6445/services/apiexport/root:e2e-org-p2l7n:workspace1/export1\"}]}}"
09:59:49.516186   30606 permissionclaimlabel_controller.go:118] "queueing APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:49.516230   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:49.516331   30606 permissionclaimlabel_reconcile.go:90] "claim set details" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" expected=map[] unexpected=map[] toApply=map[] toRemove=map[] all=map[]
09:59:49.525283   30606 apibinding_controller.go:260] "queueing APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:49.666497   30606 apibinding_controller.go:260] "queueing APIBinding because of APIExport" reconciler="kcp-apibinding" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:49.666543   30606 apiexport_controller.go:212] "queueing APIExport" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:49.667011   30606 apiexport_apireconciler_controller.go:172] "queueing APIExport" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:49.667354   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:49.667625   30606 apiexport_apireconciler_controller.go:220] "processing key" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:49.667689   30606 apiexport_apireconciler_reconcile.go:211] "creating API definition" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1" apiDomainKey=root:e2e-org-p2l7n:workspace1/export1 apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" gvr="example.io/v1, Resource=widgets" labels=[]
09:59:49.823868   30606 apibinding_controller.go:372] "processing key" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:49.824105   30606 committer.go:113] "patching *v1alpha1.APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" patch="{\"metadata\":{\"resourceVersion\":\"160139\",\"uid\":\"6a3dddf3-5b8a-4563-b1db-5c149dd936bd\"},\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2022-10-25T09:59:49Z\",\"message\":\"Waiting for API(s) to be established\",\"reason\":\"WaitingForEstablished\",\"severity\":\"Info\",\"status\":\"False\",\"type\":\"Ready\"},{\"lastTransitionTime\":\"2022-10-25T09:59:49Z\",\"message\":\"Waiting for API(s) to be established\",\"reason\":\"WaitingForEstablished\",\"severity\":\"Info\",\"status\":\"False\",\"type\":\"InitialBindingCompleted\"},{\"lastTransitionTime\":\"2022-10-25T09:59:49Z\",\"status\":\"True\",\"type\":\"PermissionClaimsApplied\"},{\"lastTransitionTime\":\"2022-10-25T09:59:49Z\",\"status\":\"True\",\"type\":\"PermissionClaimsValid\"}],\"phase\":\"Binding\"}}"
09:59:49.862603   30606 apiexport_apireconciler_reconcile.go:251] "updating APIs" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1" apiDomainKey=root:e2e-org-p2l7n:workspace1/export1 apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" new=[widgets.v1.example.io apibindings.v1alpha1.apis.kcp.dev] preserved=[] removed=[]
09:59:50.023237   30606 apibinding_controller.go:260] "queueing APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:50.023377   30606 permissionclaimlabel_controller.go:118] "queueing APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:50.023434   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:50.023550   30606 permissionclaimlabel_reconcile.go:90] "claim set details" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" expected=map[] unexpected=map[] toApply=map[] toRemove=map[] all=map[]
09:59:50.500835   30606 apibinding_controller.go:372] "processing key" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:50.501075   30606 committer.go:113] "patching *v1alpha1.APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" patch="{\"metadata\":{\"resourceVersion\":\"160152\",\"uid\":\"6a3dddf3-5b8a-4563-b1db-5c149dd936bd\"},\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2022-10-25T09:59:50Z\",\"message\":\"Waiting for API(s) to be established\",\"reason\":\"WaitingForEstablished\",\"severity\":\"Info\",\"status\":\"False\",\"type\":\"Ready\"},{\"lastTransitionTime\":\"2022-10-25T09:59:50Z\",\"message\":\"Waiting for API(s) to be established\",\"reason\":\"WaitingForEstablished\",\"severity\":\"Info\",\"status\":\"False\",\"type\":\"InitialBindingCompleted\"},{\"lastTransitionTime\":\"2022-10-25T09:59:49Z\",\"status\":\"True\",\"type\":\"PermissionClaimsApplied\"},{\"lastTransitionTime\":\"2022-10-25T09:59:49Z\",\"status\":\"True\",\"type\":\"PermissionClaimsValid\"}],\"phase\":\"Binding\"}}"
09:59:50.836464   30606 apibinding_controller.go:260] "queueing APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:50.836598   30606 permissionclaimlabel_controller.go:118] "queueing APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:50.836655   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:50.836752   30606 permissionclaimlabel_reconcile.go:90] "claim set details" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" expected=map[] unexpected=map[] toApply=map[] toRemove=map[] all=map[]
09:59:51.077196   30606 apibinding_controller.go:372] "processing key" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:51.077521   30606 apibinding_reconcile.go:273] "creating CRD" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" apiresourceschema.workspace="root:e2e-org-p2l7n:workspace1" apiresourceschema.namespace="" apiresourceschema.name="v1.widgets.example.io" apiresourceschema.apiVersion="" customresourcedefinition.workspace="system:bound-crds" customresourcedefinition.namespace="" customresourcedefinition.name="098c4b2c-e025-4339-a60b-5d91344a52fa" customresourcedefinition.apiVersion="" groupResource="widgets.example.io"
09:59:51.497023   30606 apibinding_controller.go:260] "queueing APIBinding because of APIExport because of APIResourceSchema because of CRD" reconciler="kcp-apibinding" customresourcedefinition.workspace="system:bound-crds" customresourcedefinition.namespace="" customresourcedefinition.name="098c4b2c-e025-4339-a60b-5d91344a52fa" customresourcedefinition.apiVersion="" groupResource="widgets.example.io" established=false apiresourceschema.workspace="root:e2e-org-p2l7n:workspace1" apiresourceschema.namespace="" apiresourceschema.name="v1.widgets.example.io" apiresourceschema.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:51.497665   30606 committer.go:113] "patching *v1alpha1.APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" patch="{\"metadata\":{\"resourceVersion\":\"160171\",\"uid\":\"6a3dddf3-5b8a-4563-b1db-5c149dd936bd\"},\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2022-10-25T09:59:51Z\",\"message\":\"Waiting for API(s) to be established: v1.widgets.example.io\",\"reason\":\"WaitingForEstablished\",\"severity\":\"Info\",\"status\":\"False\",\"type\":\"Ready\"},{\"lastTransitionTime\":\"2022-10-25T09:59:51Z\",\"status\":\"True\",\"type\":\"APIExportValid\"},{\"lastTransitionTime\":\"2022-10-25T09:59:51Z\",\"message\":\"Waiting for API(s) to be established: v1.widgets.example.io\",\"reason\":\"WaitingForEstablished\",\"severity\":\"Info\",\"status\":\"False\",\"type\":\"BindingUpToDate\"},{\"lastTransitionTime\":\"2022-10-25T09:59:51Z\",\"message\":\"Waiting for API(s) to be established: v1.widgets.example.io\",\"reason\":\"WaitingForEstablished\",\"severity\":\"Info\",\"status\":\"False\",\"type\":\"InitialBindingCompleted\"},{\"lastTransitionTime\":\"2022-10-25T09:59:49Z\",\"status\":\"True\",\"type\":\"PermissionClaimsApplied\"},{\"lastTransitionTime\":\"2022-10-25T09:59:49Z\",\"status\":\"True\",\"type\":\"PermissionClaimsValid\"}]}}"
09:59:51.589540   30606 apibinding_controller.go:260] "queueing APIBinding because of APIExport because of APIResourceSchema because of CRD" reconciler="kcp-apibinding" customresourcedefinition.workspace="system:bound-crds" customresourcedefinition.namespace="" customresourcedefinition.name="098c4b2c-e025-4339-a60b-5d91344a52fa" customresourcedefinition.apiVersion="" groupResource="widgets.example.io" established=false apiresourceschema.workspace="root:e2e-org-p2l7n:workspace1" apiresourceschema.namespace="" apiresourceschema.name="v1.widgets.example.io" apiresourceschema.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:51.690763   30606 apibinding_controller.go:260] "queueing APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:51.690823   30606 permissionclaimlabel_controller.go:118] "queueing APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:51.690865   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:51.691018   30606 permissionclaimlabel_reconcile.go:90] "claim set details" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" expected=map[] unexpected=map[] toApply=map[] toRemove=map[] all=map[]
09:59:51.752509   30606 apibinding_controller.go:260] "queueing APIBinding because of APIExport because of APIResourceSchema because of CRD" reconciler="kcp-apibinding" customresourcedefinition.workspace="system:bound-crds" customresourcedefinition.namespace="" customresourcedefinition.name="098c4b2c-e025-4339-a60b-5d91344a52fa" customresourcedefinition.apiVersion="" groupResource="widgets.example.io" established=true apiresourceschema.workspace="root:e2e-org-p2l7n:workspace1" apiresourceschema.namespace="" apiresourceschema.name="v1.widgets.example.io" apiresourceschema.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:52.167390   30606 apibinding_controller.go:372] "processing key" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:52.167750   30606 committer.go:113] "patching *v1alpha1.APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" patch="{\"metadata\":{\"resourceVersion\":\"160205\",\"uid\":\"6a3dddf3-5b8a-4563-b1db-5c149dd936bd\"},\"status\":{\"boundResources\":[{\"group\":\"example.io\",\"resource\":\"widgets\",\"schema\":{\"UID\":\"098c4b2c-e025-4339-a60b-5d91344a52fa\",\"identityHash\":\"a81718b4324e2c8626859fef2d6e18159cb13117bc861df955b09310d0468335\",\"name\":\"v1.widgets.example.io\"},\"storageVersions\":[\"v1\"]}],\"conditions\":[{\"lastTransitionTime\":\"2022-10-25T09:59:52Z\",\"status\":\"True\",\"type\":\"Ready\"},{\"lastTransitionTime\":\"2022-10-25T09:59:51Z\",\"status\":\"True\",\"type\":\"APIExportValid\"},{\"lastTransitionTime\":\"2022-10-25T09:59:52Z\",\"status\":\"True\",\"type\":\"BindingUpToDate\"},{\"lastTransitionTime\":\"2022-10-25T09:59:52Z\",\"status\":\"True\",\"type\":\"InitialBindingCompleted\"},{\"lastTransitionTime\":\"2022-10-25T09:59:49Z\",\"status\":\"True\",\"type\":\"PermissionClaimsApplied\"},{\"lastTransitionTime\":\"2022-10-25T09:59:49Z\",\"status\":\"True\",\"type\":\"PermissionClaimsValid\"}],\"phase\":\"Bound\"}}"
09:59:52.341330   30606 apiexport_controller.go:279] "queueing APIExport via APIBinding" reconciler="kcp-apiexport" apibinding.workspace="root:e2e-org-p2l7n:workspace2" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:52.341354   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:52.465478   30606 apibinding_controller.go:260] "queueing APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:52.465544   30606 apibinding_controller.go:372] "processing key" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:52.465906   30606 permissionclaimlabel_controller.go:118] "queueing APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:52.465993   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:52.466064   30606 permissionclaimlabel_reconcile.go:90] "claim set details" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" expected=map[] unexpected=map[] toApply=map[] toRemove=map[] all=map[]
09:59:52.466160   30606 apiexport_controller.go:279] "queueing APIExport via APIBinding" reconciler="kcp-apiexport" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:52.466212   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:53.956320   30606 apiexport_apireconciler_controller.go:172] "queueing APIExport" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:53.957318   30606 apiexport_apireconciler_controller.go:220] "processing key" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:53.957435   30606 apiexport_apireconciler_reconcile.go:211] "creating API definition" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1" apiDomainKey=root:e2e-org-p2l7n:workspace1/export1 apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" gvr="apis.kcp.dev/v1alpha1, Resource=apibindings" labels=[{key:claimed.internal.apis.kcp.dev/4GyTEaSaPpWG6HDgrycl3WQoTcFfH3v0JXrUss operator:in strValues:[6a124Qwo5jLqGSOGhnYPh6CjwqWTIQDVgyMLKn LstvmbbzVDDOn90ZbhzSQO5U3DMCf88h1pZ]}]
09:59:53.957012   30606 apibinding_controller.go:260] "queueing APIBinding because of APIExport" reconciler="kcp-apibinding" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:53.957028   30606 apiexport_controller.go:212] "queueing APIExport" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:53.964759   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:53.984243   30606 apiexport_apireconciler_reconcile.go:251] "updating APIs" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1" apiDomainKey=root:e2e-org-p2l7n:workspace1/export1 apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" new=[apibindings.v1alpha1.apis.kcp.dev] preserved=[widgets.v1.example.io] removed=[apibindings.v1alpha1.apis.kcp.dev]
09:59:54.125860   30606 apibinding_controller.go:372] "processing key" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:54.126249   30606 committer.go:113] "patching *v1alpha1.APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" patch="{\"metadata\":{\"resourceVersion\":\"160226\",\"uid\":\"6a3dddf3-5b8a-4563-b1db-5c149dd936bd\"},\"status\":{\"exportPermissionClaims\":[{\"group\":\"apis.kcp.dev\",\"resource\":\"apibindings\"}]}}"
09:59:54.361589   30606 apibinding_controller.go:260] "queueing APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:54.361826   30606 permissionclaimlabel_controller.go:118] "queueing APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:54.361978   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:54.362106   30606 permissionclaimlabel_reconcile.go:90] "claim set details" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" expected=map[] unexpected=map[] toApply=map[] toRemove=map[] all=map[]
09:59:54.363139   30606 apiexport_controller.go:279] "queueing APIExport via APIBinding" reconciler="kcp-apiexport" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:54.363215   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:54.380915   30606 apiexport_controller.go:279] "queueing APIExport via APIBinding" reconciler="kcp-apiexport" apibinding.workspace="root:e2e-org-p2l7n:workspace2" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:54.387051   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:54.555788   30606 apibinding_controller.go:260] "queueing APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:54.555861   30606 permissionclaimlabel_controller.go:118] "queueing APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:54.555951   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:54.556071   30606 permissionclaimlabel_reconcile.go:90] "claim set details" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" expected=map[apibindings/apis.kcp.dev/:{}] unexpected=map[] toApply=map[apibindings/apis.kcp.dev/:{}] toRemove=map[] all=map[apibindings/apis.kcp.dev/:{}]
09:59:54.556207   30606 permissionclaimlabel_reconcile.go:114] "listing resources" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" claim="apibindings/apis.kcp.dev/"
09:59:54.556288   30606 permissionclaimlabel_reconcile.go:124] "got resources" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" claim="apibindings/apis.kcp.dev/" count=6
09:59:54.556367   30606 permissionclaimlabel_reconcile.go:143] "patching to get claim labels updated" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" partialobjectmetadata.workspace="root:e2e-org-p2l7n:workspace1" partialobjectmetadata.namespace="" partialobjectmetadata.name="binding2" partialobjectmetadata.apiVersion="meta.k8s.io/v1"
09:59:54.556596   30606 apiexport_controller.go:279] "queueing APIExport via APIBinding" reconciler="kcp-apiexport" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:54.557158   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:54.590628   30606 apibinding_controller.go:372] "processing key" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:54.837661   30606 permissionclaimlabel_reconcile.go:143] "patching to get claim labels updated" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" partialobjectmetadata.workspace="root:e2e-org-p2l7n:workspace1" partialobjectmetadata.namespace="" partialobjectmetadata.name="tenancy.kcp.dev-6vuyx" partialobjectmetadata.apiVersion="meta.k8s.io/v1"
09:59:54.852664   30606 permissionclaimlabel_controller.go:118] "queueing APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:54.852902   30606 apiexport_controller.go:279] "queueing APIExport via APIBinding" reconciler="kcp-apiexport" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:54.852946   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:54.859743   30606 apibinding_controller.go:260] "queueing APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:54.997520   30606 permissionclaimlabel_reconcile.go:143] "patching to get claim labels updated" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" partialobjectmetadata.workspace="root:e2e-org-p2l7n:workspace1" partialobjectmetadata.namespace="" partialobjectmetadata.name="scheduling.kcp.dev-1epai" partialobjectmetadata.apiVersion="meta.k8s.io/v1"
09:59:55.000981   30606 apibinding_controller.go:372] "processing key" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:55.161603   30606 permissionclaimlabel_reconcile.go:143] "patching to get claim labels updated" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" partialobjectmetadata.workspace="root:e2e-org-p2l7n:workspace1" partialobjectmetadata.namespace="" partialobjectmetadata.name="workload.kcp.dev-4oesy" partialobjectmetadata.apiVersion="meta.k8s.io/v1"
09:59:55.289600   30606 permissionclaimlabel_reconcile.go:143] "patching to get claim labels updated" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" partialobjectmetadata.workspace="root:e2e-org-p2l7n:workspace1" partialobjectmetadata.namespace="" partialobjectmetadata.name="apiresource.kcp.dev-bffkd" partialobjectmetadata.apiVersion="meta.k8s.io/v1"
09:59:55.387273   30606 permissionclaimlabel_controller.go:221] "patching APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" patch="{\"metadata\":{\"resourceVersion\":\"160308\",\"uid\":\"6a3dddf3-5b8a-4563-b1db-5c149dd936bd\"},\"status\":{\"appliedPermissionClaims\":[{\"group\":\"apis.kcp.dev\",\"resource\":\"apibindings\"}]}}"
09:59:55.452168   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:55.452256   30606 permissionclaimlabel_reconcile.go:90] "claim set details" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" expected=map[apibindings/apis.kcp.dev/:{}] unexpected=map[] toApply=map[apibindings/apis.kcp.dev/:{}] toRemove=map[] all=map[apibindings/apis.kcp.dev/:{}]
09:59:55.452379   30606 permissionclaimlabel_reconcile.go:114] "listing resources" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" claim="apibindings/apis.kcp.dev/"
09:59:55.452436   30606 permissionclaimlabel_reconcile.go:124] "got resources" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" claim="apibindings/apis.kcp.dev/" count=6
09:59:55.452474   30606 permissionclaimlabel_reconcile.go:143] "patching to get claim labels updated" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" partialobjectmetadata.workspace="root:e2e-org-p2l7n:workspace1" partialobjectmetadata.namespace="" partialobjectmetadata.name="binding2" partialobjectmetadata.apiVersion="meta.k8s.io/v1"
09:59:55.538498   30606 permissionclaimlabel_reconcile.go:143] "patching to get claim labels updated" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" partialobjectmetadata.workspace="root:e2e-org-p2l7n:workspace1" partialobjectmetadata.namespace="" partialobjectmetadata.name="tenancy.kcp.dev-6vuyx" partialobjectmetadata.apiVersion="meta.k8s.io/v1"
09:59:55.626716   30606 permissionclaimlabel_reconcile.go:143] "patching to get claim labels updated" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" partialobjectmetadata.workspace="root:e2e-org-p2l7n:workspace1" partialobjectmetadata.namespace="" partialobjectmetadata.name="scheduling.kcp.dev-1epai" partialobjectmetadata.apiVersion="meta.k8s.io/v1"
09:59:55.632311   30606 apibinding_controller.go:260] "queueing APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:55.632381   30606 apibinding_controller.go:372] "processing key" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:55.632745   30606 permissionclaimlabel_controller.go:118] "queueing APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:55.632828   30606 apiexport_controller.go:279] "queueing APIExport via APIBinding" reconciler="kcp-apiexport" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:55.632913   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:55.739201   30606 permissionclaimlabel_reconcile.go:143] "patching to get claim labels updated" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" partialobjectmetadata.workspace="root:e2e-org-p2l7n:workspace1" partialobjectmetadata.namespace="" partialobjectmetadata.name="workload.kcp.dev-4oesy" partialobjectmetadata.apiVersion="meta.k8s.io/v1"
09:59:55.916691   30606 permissionclaimlabel_reconcile.go:143] "patching to get claim labels updated" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" partialobjectmetadata.workspace="root:e2e-org-p2l7n:workspace1" partialobjectmetadata.namespace="" partialobjectmetadata.name="apiresource.kcp.dev-bffkd" partialobjectmetadata.apiVersion="meta.k8s.io/v1"
09:59:56.024916   30606 permissionclaimlabel_controller.go:221] "patching APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" patch="{\"metadata\":{\"resourceVersion\":\"160320\",\"uid\":\"6a3dddf3-5b8a-4563-b1db-5c149dd936bd\"},\"status\":{\"appliedPermissionClaims\":[{\"group\":\"apis.kcp.dev\",\"resource\":\"apibindings\"}]}}"
09:59:56.077445   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:56.077550   30606 permissionclaimlabel_reconcile.go:90] "claim set details" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" expected=map[] unexpected=map[] toApply=map[] toRemove=map[] all=map[]
09:59:56.090327   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:56.090474   30606 permissionclaimlabel_reconcile.go:90] "claim set details" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" expected=map[] unexpected=map[] toApply=map[] toRemove=map[] all=map[]
10:00:26.055231   30606 apiexport_controller.go:279] "queueing APIExport via APIBinding" reconciler="kcp-apiexport" apibinding.workspace="root:e2e-org-p2l7n:workspace2" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" key="root:e2e-org-p2l7n:workspace1|export1"
10:00:26.056008   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
10:00:27.425613   30606 apiexport_controller.go:279] "queueing APIExport via APIBinding" reconciler="kcp-apiexport" apibinding.workspace="root:e2e-org-p2l7n:workspace2" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" key="root:e2e-org-p2l7n:workspace1|export1"
10:00:27.425666   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
10:00:27.457108   30606 apiexport_apireconciler_controller.go:172] "queueing APIExport" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1"
10:00:27.457292   30606 apiexport_apireconciler_controller.go:220] "processing key" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1"
10:00:27.457390   30606 apiexport_apireconciler_reconcile.go:60] "deleting APIs for API domain key" reconciler="kcp-virtual-apiexport-api-reconciler" key="root:e2e-org-p2l7n:workspace1|export1" apiDomainKey=root:e2e-org-p2l7n:workspace1/export1
10:00:27.460584   30606 apibinding_controller.go:260] "queueing APIBinding because of APIExport" reconciler="kcp-apibinding" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:27.460688   30606 apibinding_controller.go:372] "processing key" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:27.461131   30606 committer.go:113] "patching *v1alpha1.APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" patch="{\"metadata\":{\"resourceVersion\":\"160360\",\"uid\":\"6a3dddf3-5b8a-4563-b1db-5c149dd936bd\"},\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2022-10-25T09:59:52Z\",\"status\":\"True\",\"type\":\"Ready\"},{\"lastTransitionTime\":\"2022-10-25T10:00:27Z\",\"message\":\"APIExport root:e2e-org-p2l7n:workspace1|export1 not found\",\"reason\":\"APIExportNotFound\",\"severity\":\"Error\",\"status\":\"False\",\"type\":\"APIExportValid\"},{\"lastTransitionTime\":\"2022-10-25T09:59:52Z\",\"status\":\"True\",\"type\":\"BindingUpToDate\"},{\"lastTransitionTime\":\"2022-10-25T09:59:52Z\",\"status\":\"True\",\"type\":\"InitialBindingCompleted\"},{\"lastTransitionTime\":\"2022-10-25T09:59:49Z\",\"status\":\"True\",\"type\":\"PermissionClaimsApplied\"},{\"lastTransitionTime\":\"2022-10-25T09:59:49Z\",\"status\":\"True\",\"type\":\"PermissionClaimsValid\"}]}}"
10:00:27.461444   30606 apiexport_controller.go:212] "queueing APIExport" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
10:00:27.461521   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
10:00:27.624495   30606 apibinding_controller.go:260] "queueing APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:27.625270   30606 apibinding_controller.go:372] "processing key" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:27.625583   30606 permissionclaimlabel_controller.go:118] "queueing APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:27.625641   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:27.625839   30606 permissionclaimlabel_reconcile.go:61] "error getting APIExport" err="apiexport.apis.kcp.dev \"root:e2e-org-p2l7n:workspace1|export1\" not found" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiExportWorkspace="root:e2e-org-p2l7n:workspace1" apiExportName="export1"
10:00:27.625981   30606 apiexport_controller.go:279] "queueing APIExport via APIBinding" reconciler="kcp-apiexport" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" key="root:e2e-org-p2l7n:workspace1|export1"
10:00:27.630961   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
10:00:28.098200   30606 apiexport_controller.go:279] "queueing APIExport via APIBinding" reconciler="kcp-apiexport" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" key="root:e2e-org-p2l7n:workspace1|export1"
10:00:28.098294   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
10:00:28.100728   30606 apibinding_controller.go:260] "queueing APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:28.100767   30606 apibinding_controller.go:372] "processing key" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:28.105924   30606 permissionclaimlabel_controller.go:118] "queueing APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:28.105983   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:28.106057   30606 permissionclaimlabel_reconcile.go:61] "error getting APIExport" err="apiexport.apis.kcp.dev \"root:e2e-org-p2l7n:workspace1|export1\" not found" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiExportWorkspace="root:e2e-org-p2l7n:workspace1" apiExportName="export1"
10:00:30.948535   30606 apibinding_controller.go:260] "queueing APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:30.948581   30606 permissionclaimlabel_controller.go:118] "queueing APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:30.948607   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:30.948703   30606 permissionclaimlabel_reconcile.go:61] "error getting APIExport" err="apiexport.apis.kcp.dev \"root:e2e-org-p2l7n:workspace1|export1\" not found" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiExportWorkspace="root:e2e-org-p2l7n:workspace1" apiExportName="export1"
10:00:30.948794   30606 apibinding_deletion_controller.go:134] "queueing APIBinding" reconciler="kcp-apibindingdeletion" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:30.949236   30606 apibinding_deletion_controller.go:169] "processing key" reconciler="kcp-apibindingdeletion" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:30.949490   30606 apiexport_controller.go:279] "queueing APIExport via APIBinding" reconciler="kcp-apiexport" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" key="root:e2e-org-p2l7n:workspace1|export1"
10:00:30.959870   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
10:00:31.060600   30606 apibinding_deletion_controller.go:269] "finalizing APIBinding" reconciler="kcp-apibindingdeletion" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion=""
10:00:31.061333   30606 committer.go:113] "patching *v1alpha1.APIBinding" reconciler="kcp-apibindingdeletion" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" patch="{\"metadata\":{\"finalizers\":null,\"resourceVersion\":\"161207\",\"uid\":\"6a3dddf3-5b8a-4563-b1db-5c149dd936bd\"}}"
10:00:31.357244   30606 apibinding_deletion_controller.go:208] "finished syncing" reconciler="kcp-apibindingdeletion" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" duration="407.971162ms"
10:00:31.379032   30606 apibinding_controller.go:260] "queueing APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:31.379182   30606 permissionclaimlabel_controller.go:118] "queueing APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:31.379262   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
10:00:31.379339   30606 apiexport_controller.go:279] "queueing APIExport via APIBinding" reconciler="kcp-apiexport" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" key="root:e2e-org-p2l7n:workspace1|export1"
10:00:31.379418   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
10:00:31.518870   30606 apibinding_controller.go:372] "processing key" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"

stevekuznetsov avatar Oct 26 '22 12:10 stevekuznetsov

@ncdc looks like it's expected=map[] unexpected=map[] toApply=map[] toRemove=map[] all=map[] every time :facepalm:

stevekuznetsov avatar Oct 26 '22 12:10 stevekuznetsov

We've got the test clearing the accepted claims from spec:

$ grep -h 6dcf1a1f-3aa2-4d9e-af24-b96006926f7c e2e-sharded/sharded/artifacts/audit-* | jq .requestObject
{
  "spec": {
    "permissionClaims": null
  }
}

Then, the responses ...

09:59:55.610257   30606 httplog.go:131] "HTTP" verb="PATCH" URI="/clusters/root:e2e-org-p2l7n:workspace1/apis/apis.kcp.dev/v1alpha1/apibindings/binding1" latency="56.293112ms" userAgent="apiexport.test/v0.0.0 (linux/amd64) kubernetes/$Format" audit-ID="6dcf1a1f-3aa2-4d9e-af24-b96006926f7c" srcIP="10.1.0.57:59142" resp=200
09:59:55.614483   30606 httplog.go:131] "HTTP" verb="PATCH" URI="/clusters/root:e2e-org-p2l7n:workspace1/apis/apis.kcp.dev/v1alpha1/apibindings/tenancy.kcp.dev-6vuyx" latency="69.701863ms" userAgent="kcp/v1.24.3+kcp (linux/amd64) kubernetes/c62bf37/kcp-permissionclaimlabel" audit-ID="8f6bef10-d872-4b6d-8169-f44c0963e95e" srcIP="[::1]:52080" resp=200
09:59:55.626716   30606 permissionclaimlabel_reconcile.go:143] "patching to get claim labels updated" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" partialobjectmetadata.workspace="root:e2e-org-p2l7n:workspace1" partialobjectmetadata.namespace="" partialobjectmetadata.name="scheduling.kcp.dev-1epai" partialobjectmetadata.apiVersion="meta.k8s.io/v1"
09:59:55.632311   30606 apibinding_controller.go:260] "queueing APIBinding" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:55.632381   30606 apibinding_controller.go:372] "processing key" reconciler="kcp-apibinding" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:55.632745   30606 permissionclaimlabel_controller.go:118] "queueing APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:55.632828   30606 apiexport_controller.go:279] "queueing APIExport via APIBinding" reconciler="kcp-apiexport" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:55.632913   30606 apiexport_controller.go:315] "processing key" reconciler="kcp-apiexport" key="root:e2e-org-p2l7n:workspace1|export1"
09:59:55.718770   30606 httplog.go:131] "HTTP" verb="PATCH" URI="/clusters/root:e2e-org-p2l7n:workspace1/apis/apis.kcp.dev/v1alpha1/apibindings/scheduling.kcp.dev-1epai" latency="86.043024ms" userAgent="kcp/v1.24.3+kcp (linux/amd64) kubernetes/c62bf37/kcp-permissionclaimlabel" audit-ID="b24eebdc-91e7-4b66-963a-53bcd5f9e9d9" srcIP="[::1]:52080" resp=200
09:59:55.739201   30606 permissionclaimlabel_reconcile.go:143] "patching to get claim labels updated" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" partialobjectmetadata.workspace="root:e2e-org-p2l7n:workspace1" partialobjectmetadata.namespace="" partialobjectmetadata.name="workload.kcp.dev-4oesy" partialobjectmetadata.apiVersion="meta.k8s.io/v1"
09:59:55.776135   30606 httplog.go:131] "HTTP" verb="POST" URI="/clusters/root:e2e-org-p2l7n:workspace1/apis/authorization.k8s.io/v1/subjectaccessreviews" latency="3.928414ms" userAgent="kcp/v1.24.3+kcp (linux/amd64) kubernetes/c62bf37/apiexport-virtual-workspace" audit-ID="a9d1db4f-0c1b-4614-844d-5d5a335208e9" srcIP="[::1]:52080" resp=201
09:59:55.880521   30606 httplog.go:131] "HTTP" verb="PATCH" URI="/clusters/root:e2e-org-p2l7n:workspace1/apis/apis.kcp.dev/v1alpha1/apibindings/workload.kcp.dev-4oesy" latency="122.198861ms" userAgent="kcp/v1.24.3+kcp (linux/amd64) kubernetes/c62bf37/kcp-permissionclaimlabel" audit-ID="04fc7063-bbf0-419c-944e-5a77748c7b09" srcIP="[::1]:52080" resp=200
09:59:55.916691   30606 permissionclaimlabel_reconcile.go:143] "patching to get claim labels updated" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" partialobjectmetadata.workspace="root:e2e-org-p2l7n:workspace1" partialobjectmetadata.namespace="" partialobjectmetadata.name="apiresource.kcp.dev-bffkd" partialobjectmetadata.apiVersion="meta.k8s.io/v1"
09:59:55.927975   30606 httplog.go:131] "HTTP" verb="LIST" URI="/services/apiexport/root:e2e-org-p2l7n:workspace1/export1/clusters/%2A/apis/apis.kcp.dev/v1alpha1/apibindings" latency="186.905305ms" userAgent="apiexport.test/v0.0.0 (linux/amd64) kubernetes/$Format" audit-ID="0bcb20d1-0d4c-408b-a96c-9c3ab2f71f96" srcIP="10.1.0.57:39792" resp=200
09:59:55.971517   30606 httplog.go:131] "HTTP" verb="POST" URI="/clusters/root:e2e-org-p2l7n:workspace1/apis/authorization.k8s.io/v1/subjectaccessreviews" latency="1.768207ms" userAgent="kcp/v1.24.3+kcp (linux/amd64) kubernetes/c62bf37/apiexport-virtual-workspace" audit-ID="a295f4ff-14ca-4869-bb52-77a4292732b6" srcIP="[::1]:52080" resp=201
09:59:56.006546   30606 httplog.go:131] "HTTP" verb="PATCH" URI="/clusters/root:e2e-org-p2l7n:workspace1/apis/apis.kcp.dev/v1alpha1/apibindings/apiresource.kcp.dev-bffkd" latency="79.966102ms" userAgent="kcp/v1.24.3+kcp (linux/amd64) kubernetes/c62bf37/kcp-permissionclaimlabel" audit-ID="fd918f21-80ba-44be-a05e-2cd0127bb7f4" srcIP="[::1]:52080" resp=200
09:59:56.024916   30606 permissionclaimlabel_controller.go:221] "patching APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" patch="{\"metadata\":{\"resourceVersion\":\"160320\",\"uid\":\"6a3dddf3-5b8a-4563-b1db-5c149dd936bd\"},\"status\":{\"appliedPermissionClaims\":[{\"group\":\"apis.kcp.dev\",\"resource\":\"apibindings\"}]}}"
09:59:56.055517   30606 httplog.go:131] "HTTP" verb="PATCH" URI="/clusters/root:e2e-org-p2l7n:workspace1/apis/apis.kcp.dev/v1alpha1/apibindings/binding1/status" latency="25.753197ms" userAgent="kcp/v1.24.3+kcp (linux/amd64) kubernetes/c62bf37/kcp-permissionclaimlabel" audit-ID="2bd04461-2722-4750-8ef1-dfdfa9fbfa9c" srcIP="[::1]:52080" resp=409
09:59:56.077445   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:56.077550   30606 permissionclaimlabel_reconcile.go:90] "claim set details" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" expected=map[] unexpected=map[] toApply=map[] toRemove=map[] all=map[]
09:59:56.090327   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:56.090474   30606 permissionclaimlabel_reconcile.go:90] "claim set details" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" expected=map[] unexpected=map[] toApply=map[] toRemove=map[] all=map[]
09:59:56.131483   30606 httplog.go:131] "HTTP" verb="LIST" URI="/services/apiexport/root:e2e-org-p2l7n:workspace1/export1/clusters/%2A/apis/apis.kcp.dev/v1alpha1/apibindings" latency="166.463228ms" userAgent="apiexport.test/v0.0.0 (linux/amd64) kubernetes/$Format" audit-ID="f823fc95-ad19-49c2-bc4a-3887edd14a6b" srcIP="10.1.0.57:39792" resp=200
09:59:56.153039   30606 httplog.go:131] "HTTP" verb="POST" URI="/clusters/root:e2e-org-p2l7n:workspace1/apis/authorization.k8s.io/v1/subjectaccessreviews" latency="1.888507ms" userAgent="kcp/v1.24.3+kcp (linux/amd64) kubernetes/c62bf37/apiexport-virtual-workspace" audit-ID="84073be7-5eb1-4d2a-913d-11a081ec1114" srcIP="[::1]:52080" resp=201

stevekuznetsov avatar Oct 26 '22 13:10 stevekuznetsov

reconciler="kcp-permissionclaimlabel" re-labels all of the APIBindings in the workspace, correctly, and then issues this patch to the APIBinding in question:

$ grep -h 2bd04461-2722-4750-8ef1-dfdfa9fbfa9c /home/stevekuznetsov/Downloads/e2e-sharded/sharded/artifacts/audit-* | jq .requestObject
{
  "metadata": {
    "resourceVersion": "160320",
    "uid": "6a3dddf3-5b8a-4563-b1db-5c149dd936bd"
  },
  "status": {
    "appliedPermissionClaims": [
      {
        "group": "apis.kcp.dev",
        "resource": "apibindings"
      }
    ]
  }
}

This gets the 409 and then on re-queue fails to do anything with it ...

stevekuznetsov avatar Oct 26 '22 13:10 stevekuznetsov

Ah, sorry - so that is the controller processing the previous case, where we added APIBindings:

09:59:54.555861   30606 permissionclaimlabel_controller.go:118] "queueing APIBinding" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:54.555951   30606 permissionclaimlabel_controller.go:154] "processing key" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1"
09:59:54.556071   30606 permissionclaimlabel_reconcile.go:90] "claim set details" reconciler="kcp-permissionclaimlabel" key="root:e2e-org-p2l7n:workspace1|binding1" apibinding.workspace="root:e2e-org-p2l7n:workspace1" apibinding.namespace="" apibinding.name="binding1" apibinding.apiVersion="" apiexport.workspace="root:e2e-org-p2l7n:workspace1" apiexport.namespace="" apiexport.name="export1" apiexport.apiVersion="" expected=map[apibindings/apis.kcp.dev/:{}] unexpected=map[] toApply=map[apibindings/apis.kcp.dev/:{}] toRemove=map[] all=map[apibindings/apis.kcp.dev/:{}]

In the middle of that flow, we remove the permission claim, while it's still labeling things. Then we end up attempting to patch status, but get the 409, so we never exposed that we were labeling it in the positive direction, and on the re-queue, we have no permission claims in spec (since the test client removed it) and nothing in status (since we failed to update that we were labelling).

stevekuznetsov avatar Oct 26 '22 13:10 stevekuznetsov

Marking the test as to-skip in https://github.com/kcp-dev/kcp/pull/2260/commits/13e6bf5862e5f69b91a01e22fe2ab4791d6117f8

stevekuznetsov avatar Oct 26 '22 13:10 stevekuznetsov

@mjudeikis can we reopen this? The skip is still there, I wonder if we should revisit fixing/changing the test.

embik avatar Dec 06 '24 08:12 embik

/reopen easy, this was cleaning and some mistakes might be there :)

mjudeikis avatar Dec 06 '24 09:12 mjudeikis

Issues go stale after 90d of inactivity. After a furter 30 days, they will turn rotten. Mark the issue as fresh with /remove-lifecycle stale.

If this issue is safe to close now please do so with /close.

/lifecycle stale

kcp-ci-bot avatar Mar 06 '25 20:03 kcp-ci-bot

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

kcp-ci-bot avatar Apr 05 '25 20:04 kcp-ci-bot

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

/close

kcp-ci-bot avatar May 05 '25 20:05 kcp-ci-bot

@kcp-ci-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kcp-ci-bot avatar May 05 '25 20:05 kcp-ci-bot

/reopen /remove-lifecycle rotten

mjudeikis avatar Oct 08 '25 08:10 mjudeikis

@mjudeikis: Reopened this issue.

In response to this:

/reopen /remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kcp-ci-bot avatar Oct 08 '25 08:10 kcp-ci-bot