shell-operator
shell-operator copied to clipboard
Inconsistent behavior with combining binding contexts
According to https://github.com/flant/shell-operator/blob/master/HOOKS.md#shell-operator-lifecycle, If there is a sequence of hook executions in a queue, then hook is executed once with an array of binding contexts. However, I've noticed that this is not consistent - sometimes it is combined and tasks are dropped from the queue, other times all tasks are executed. I'm attaching hooks and logs below.
Expected behavior (what you expected to happen):
Since I want to run cleanup tasks based each resource that trigged the Deleted
event, when I run kubectl delete applications.argoproj.io aut-410 aut-369
I expected it to always run twice - one each for aut-410 and aut-369 resource.
Actual behavior (what actually happened): Sometimes it runs for only one resource and sometimes two.
Anything else we should know?:
I inspected the json file pointed to by BINDING_CONTEXT_PATH
and confirmed that it has both the objects.
Hook script
#!/usr/bin/env ruby # frozen_string_literal: true require 'json' config = { "configVersion": 'v1', "kubernetes": [ { "name": 'OnCreateDeleteArgoApp', "apiVersion": 'argoproj.io/v1alpha1', "kind": 'Application', "executeHookOnEvent": %w[ Deleted ], "labelSelector": { "matchLabels": { "app": "qa" } }, "allowFailure": true } ] } if ARGV.size >= 1 && ARGV.first == '--config' puts config.to_json else contexts = JSON.parse(File.read(ENV['BINDING_CONTEXT_PATH'])) puts "---#{contexts.length}" contexts.each do |content| binding_name = content['binding'] resource_event = content['watchEvent'] resource_name = content.dig('object', 'metadata', 'name') resource_namespace = content.dig('object', 'spec', 'destination', 'namespace') if binding_name == 'OnCreateDeleteArgoApp' if resource_event == 'Deleted' puts "---#{resource_name} was deleted from #{resource_namespace}---" end end end end
Logs when contexts are combined
fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"debug","msg":"kubernetes[0]{OnCreateDeleteArgoApp}: Deleted argocd/Application/aut-410: send KubeEvent","time":"2021-09-29T18:57:56Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"kubernetes","event.id":"27619f3d-787c-4279-8758-b18ee71ef847","level":"debug","msg":"Create tasks for 'kubernetes' event 'Event 'Deleted' for argocd/Application/aut-410'","time":"2021-09-29T18:57:56Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"kubernetes","event.id":"27619f3d-787c-4279-8758-b18ee71ef847","level":"info","msg":"queue task HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp","operator.component":"handleEvents","queue":"main","task.id":"1abf5006-ff92-45e7-9531-03e5019f3e53","time":"2021-09-29T18:57:56Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"debug","msg":"kubernetes[0]{OnCreateDeleteArgoApp}: Deleted argocd/Application/aut-369: send KubeEvent","time":"2021-09-29T18:57:56Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"kubernetes","event.id":"0e7413ad-035d-41b2-826e-8ba0eb223575","level":"debug","msg":"Create tasks for 'kubernetes' event 'Event 'Deleted' for argocd/Application/aut-369'","time":"2021-09-29T18:57:56Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"kubernetes","event.id":"0e7413ad-035d-41b2-826e-8ba0eb223575","level":"info","msg":"queue task HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp","operator.component":"handleEvents","queue":"main","task.id":"a92bf736-6276-400c-9100-2b2696036476","time":"2021-09-29T18:57:56Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"debug","msg":"queue main: get task HookRun","time":"2021-09-29T18:57:56Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"debug","msg":"queue main: tasks after wait [HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp,id=1abf5006-f], [HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp,id=a92bf736-6]","time":"2021-09-29T18:57:56Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"info","msg":"Binding contexts from 2 tasks are combined to 2 contexts. 1 tasks are dropped from queue 'main'","time":"2021-09-29T18:57:56Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"Execute hook","queue":"main","task":"HookRun","time":"2021-09-29T18:57:56Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"debug","msg":"Executing command '/hooks/integrations.rb' in '/hooks' dir","queue":"main","task":"HookRun","time":"2021-09-29T18:57:56Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"-----2","output":"stdout","queue":"main","task":"HookRun","time":"2021-09-29T18:57:57Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"---aut-410 was deleted from aut-410---","output":"stdout","queue":"main","task":"HookRun","time":"2021-09-29T18:57:57Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"Usage: \u0026{Sys:6.9ms User:106.952ms MaxRss:39592}","queue":"main","task":"HookRun","time":"2021-09-29T18:57:57Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"Hook executed successfully","queue":"main","task":"HookRun","time":"2021-09-29T18:57:57Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"debug","msg":"queue main: tasks after handle ","time":"2021-09-29T18:57:57Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"debug","msg":"queue main: wait for task, delay 0","time":"2021-09-29T18:57:57Z"}
Logs when contexts are not combined
fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"debug","msg":"kubernetes[0]{OnCreateDeleteArgoApp}: Deleted argocd/Application/aut-410: send KubeEvent","time":"2021-09-29T18:56:57Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"kubernetes","event.id":"29d0d561-c79a-4086-a42b-3c97df56ad87","level":"debug","msg":"Create tasks for 'kubernetes' event 'Event 'Deleted' for argocd/Application/aut-410'","time":"2021-09-29T18:56:57Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"kubernetes","event.id":"29d0d561-c79a-4086-a42b-3c97df56ad87","level":"info","msg":"queue task HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp","operator.component":"handleEvents","queue":"main","task.id":"e56af739-fa33-4403-8503-5ba36a6e08c1","time":"2021-09-29T18:56:57Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"debug","msg":"queue main: get task HookRun","time":"2021-09-29T18:56:57Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"debug","msg":"queue main: tasks after wait [HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp,id=e56af739-f]","time":"2021-09-29T18:56:57Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"Execute hook","queue":"main","task":"HookRun","time":"2021-09-29T18:56:57Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"debug","msg":"Executing command '/hooks/integrations.rb' in '/hooks' dir","queue":"main","task":"HookRun","time":"2021-09-29T18:56:57Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"debug","msg":"kubernetes[0]{OnCreateDeleteArgoApp}: Deleted argocd/Application/aut-369: send KubeEvent","time":"2021-09-29T18:56:57Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"kubernetes","event.id":"4000a9d8-fac6-44d4-be7e-8d7fbcd169bf","level":"debug","msg":"Create tasks for 'kubernetes' event 'Event 'Deleted' for argocd/Application/aut-369'","time":"2021-09-29T18:56:57Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"kubernetes","event.id":"4000a9d8-fac6-44d4-be7e-8d7fbcd169bf","level":"info","msg":"queue task HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp","operator.component":"handleEvents","queue":"main","task.id":"1546bd1e-c318-4db1-9074-59c204735aef","time":"2021-09-29T18:56:57Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"-----1","output":"stdout","queue":"main","task":"HookRun","time":"2021-09-29T18:56:58Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"---aut-410 was deleted from aut-410---","output":"stdout","queue":"main","task":"HookRun","time":"2021-09-29T18:56:58Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"Usage: \u0026{Sys:3.861ms User:96.526ms MaxRss:39592}","queue":"main","task":"HookRun","time":"2021-09-29T18:56:58Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"Hook executed successfully","queue":"main","task":"HookRun","time":"2021-09-29T18:56:58Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"debug","msg":"queue main: tasks after handle [HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp,id=1546bd1e-c]","time":"2021-09-29T18:56:58Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"debug","msg":"queue main: wait for task, delay 0","time":"2021-09-29T18:56:58Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"debug","msg":"queue main: get task HookRun","time":"2021-09-29T18:56:58Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"debug","msg":"queue main: tasks after wait [HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp,id=1546bd1e-c]","time":"2021-09-29T18:56:58Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"Execute hook","queue":"main","task":"HookRun","time":"2021-09-29T18:56:58Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"debug","msg":"Executing command '/hooks/integrations.rb' in '/hooks' dir","queue":"main","task":"HookRun","time":"2021-09-29T18:56:58Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"-----1","output":"stdout","queue":"main","task":"HookRun","time":"2021-09-29T18:56:58Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"---aut-369 was deleted from aut-369---","output":"stdout","queue":"main","task":"HookRun","time":"2021-09-29T18:56:58Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"Usage: \u0026{Sys:3.853ms User:88.623ms MaxRss:39592}","queue":"main","task":"HookRun","time":"2021-09-29T18:56:58Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"Hook executed successfully","queue":"main","task":"HookRun","time":"2021-09-29T18:56:58Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"debug","msg":"queue main: tasks after handle ","time":"2021-09-29T18:56:58Z"} fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"level":"debug","msg":"queue main: wait for task, delay 0","time":"2021-09-29T18:56:58Z"}
Here's a slight variation of the same hook, except that there's a failure in one of the cleanup tasks it tries to execute. Sometimes it executes only once and sometimes more than once.
Hook script 2
#!/usr/bin/env ruby # frozen_string_literal: true require 'json' config = { "configVersion": 'v1', "kubernetes": [ { "name": 'OnCreateDeleteArgoApp', "apiVersion": 'argoproj.io/v1alpha1', "kind": 'Application', "executeHookOnEvent": %w[ Deleted ], "labelSelector": { "matchLabels": { "app": "qa" } }, "allowFailure": true } ] } if ARGV.size >= 1 && ARGV.first == '--config' puts config.to_json else contexts = JSON.parse(File.read(ENV['BINDING_CONTEXT_PATH'])) contexts.each do |content| binding_name = content['binding'] resource_event = content['watchEvent'] resource_name = content.dig('object', 'metadata', 'name') resource_namespace = content.dig('object', 'spec', 'destination', 'namespace') if binding_name == 'OnCreateDeleteArgoApp' if resource_event == 'Deleted' puts "---#{resource_name} was deleted from #{resource_namespace}---" ENV['APP_SLUG'] = resource_namespace require_relative '/scripts/delete_integrations.rb' delete_integrations(secret, secret_manager,secret_name) #some cleanup task which could fail but we don't care end end end end
Logs when contexts are combined
fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"debug","msg":"kubernetes[0]{OnCreateDeleteArgoApp}: Deleted argocd/Application/aut-410: send KubeEvent","time":"2021-09-29T21:15:01Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"kubernetes","event.id":"644023ad-c867-4407-b4a3-a992c09c9329","level":"debug","msg":"Create tasks for 'kubernetes' event 'Event 'Deleted' for argocd/Application/aut-410'","time":"2021-09-29T21:15:01Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"kubernetes","event.id":"644023ad-c867-4407-b4a3-a992c09c9329","level":"info","msg":"queue task HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp","operator.component":"handleEvents","queue":"main","task.id":"506b210e-2e6a-42f6-a08a-79c46eda621a","time":"2021-09-29T21:15:01Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"debug","msg":"kubernetes[0]{OnCreateDeleteArgoApp}: Deleted argocd/Application/aut-369: send KubeEvent","time":"2021-09-29T21:15:01Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"kubernetes","event.id":"4aa38b93-af55-4795-ace5-bb9b5239513a","level":"debug","msg":"Create tasks for 'kubernetes' event 'Event 'Deleted' for argocd/Application/aut-369'","time":"2021-09-29T21:15:01Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"kubernetes","event.id":"4aa38b93-af55-4795-ace5-bb9b5239513a","level":"info","msg":"queue task HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp","operator.component":"handleEvents","queue":"main","task.id":"c5309a18-55eb-4f8a-989c-a2e7c782e019","time":"2021-09-29T21:15:01Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"debug","msg":"queue main: get task HookRun","time":"2021-09-29T21:15:01Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"debug","msg":"queue main: tasks after wait [HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp,id=506b210e-2], [HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp,id=c5309a18-5]","time":"2021-09-29T21:15:01Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"info","msg":"Binding contexts from 2 tasks are combined to 2 contexts. 1 tasks are dropped from queue 'main'","time":"2021-09-29T21:15:01Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"Execute hook","queue":"main","task":"HookRun","time":"2021-09-29T21:15:01Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"debug","msg":"Executing command '/hooks/integrations.rb' in '/hooks' dir","queue":"main","task":"HookRun","time":"2021-09-29T21:15:01Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"---aut-410 was deleted from aut-410---","output":"stdout","queue":"main","task":"HookRun","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"/usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': Secrets Manager can't find the specified secret. (Aws::SecretsManager::Errors::ResourceNotFoundException)","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/seahorse/client/plugins/request_callback.rb:71:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/seahorse/client/request.rb:72:in `send_request'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-secretsmanager-1.49.0/lib/aws-sdk-secretsmanager/client.rb:1526:in `get_secret_value'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /scripts/delete_integrations.rb:12:in `\u003ctop (required)\u003e'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /hooks/integrations.rb:39:in `require_relative'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /hooks/integrations.rb:39:in `block in \u003cmain\u003e'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /hooks/integrations.rb:30:in `each'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /hooks/integrations.rb:30:in `\u003cmain\u003e'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"Hook failed, but allowed to fail: integrations.rb FAILED: exit status 1","queue":"main","task":"HookRun","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"debug","msg":"queue main: tasks after handle ","time":"2021-09-29T21:15:07Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"debug","msg":"queue main: wait for task, delay 0","time":"2021-09-29T21:15:07Z"}
Logs when contexts are not combined
fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"debug","msg":"kubernetes[0]{OnCreateDeleteArgoApp}: Deleted argocd/Application/aut-410: send KubeEvent","time":"2021-09-29T21:13:57Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"kubernetes","event.id":"c4aaa846-0bca-4e8d-b713-6738cf78496a","level":"debug","msg":"Create tasks for 'kubernetes' event 'Event 'Deleted' for argocd/Application/aut-410'","time":"2021-09-29T21:13:57Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"kubernetes","event.id":"c4aaa846-0bca-4e8d-b713-6738cf78496a","level":"info","msg":"queue task HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp","operator.component":"handleEvents","queue":"main","task.id":"d827479a-bced-4025-a8d6-0781b0ddaeee","time":"2021-09-29T21:13:57Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"debug","msg":"queue main: get task HookRun","time":"2021-09-29T21:13:57Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"debug","msg":"queue main: tasks after wait [HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp,id=d827479a-b]","time":"2021-09-29T21:13:57Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"Execute hook","queue":"main","task":"HookRun","time":"2021-09-29T21:13:57Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"debug","msg":"Executing command '/hooks/integrations.rb' in '/hooks' dir","queue":"main","task":"HookRun","time":"2021-09-29T21:13:57Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"debug","msg":"kubernetes[0]{OnCreateDeleteArgoApp}: Deleted argocd/Application/aut-369: send KubeEvent","time":"2021-09-29T21:13:57Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"kubernetes","event.id":"1e4b7e3d-790b-4379-a064-2e0cd9f7fc67","level":"debug","msg":"Create tasks for 'kubernetes' event 'Event 'Deleted' for argocd/Application/aut-369'","time":"2021-09-29T21:13:57Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"kubernetes","event.id":"1e4b7e3d-790b-4379-a064-2e0cd9f7fc67","level":"info","msg":"queue task HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp","operator.component":"handleEvents","queue":"main","task.id":"973e03a6-8e51-47dd-8bc3-f86f6796fa1c","time":"2021-09-29T21:13:57Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"---aut-410 was deleted from aut-410---","output":"stdout","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"/usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': Secrets Manager can't find the specified secret. (Aws::SecretsManager::Errors::ResourceNotFoundException)","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/seahorse/client/plugins/request_callback.rb:71:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/seahorse/client/request.rb:72:in `send_request'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-secretsmanager-1.49.0/lib/aws-sdk-secretsmanager/client.rb:1526:in `get_secret_value'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /scripts/delete_integrations.rb:12:in `\u003ctop (required)\u003e'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /hooks/integrations.rb:39:in `require_relative'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /hooks/integrations.rb:39:in `block in \u003cmain\u003e'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /hooks/integrations.rb:30:in `each'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /hooks/integrations.rb:30:in `\u003cmain\u003e'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"Hook failed, but allowed to fail: integrations.rb FAILED: exit status 1","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"debug","msg":"queue main: tasks after handle [HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp,id=973e03a6-8]","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"debug","msg":"queue main: wait for task, delay 0","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"debug","msg":"queue main: get task HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"debug","msg":"queue main: tasks after wait [HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp,id=973e03a6-8]","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"Execute hook","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"debug","msg":"Executing command '/hooks/integrations.rb' in '/hooks' dir","queue":"main","task":"HookRun","time":"2021-09-29T21:14:03Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"---aut-369 was deleted from aut-369---","output":"stdout","queue":"main","task":"HookRun","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"/usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': Secrets Manager can't find the specified secret. (Aws::SecretsManager::Errors::ResourceNotFoundException)","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/seahorse/client/plugins/request_callback.rb:71:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.121.1/lib/seahorse/client/request.rb:72:in `send_request'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /usr/lib/ruby/gems/2.7.0/gems/aws-sdk-secretsmanager-1.49.0/lib/aws-sdk-secretsmanager/client.rb:1526:in `get_secret_value'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /scripts/delete_integrations.rb:12:in `\u003ctop (required)\u003e'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /hooks/integrations.rb:39:in `require_relative'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /hooks/integrations.rb:39:in `block in \u003cmain\u003e'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /hooks/integrations.rb:30:in `each'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"\tfrom /hooks/integrations.rb:30:in `\u003cmain\u003e'","output":"stderr","queue":"main","task":"HookRun","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"Hook failed, but allowed to fail: integrations.rb FAILED: exit status 1","queue":"main","task":"HookRun","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"debug","msg":"queue main: tasks after handle ","time":"2021-09-29T21:14:09Z"} fit-qa-integrations-588c8bfb76-gznnv fit-qa-integrations {"level":"debug","msg":"queue main: wait for task, delay 0","time":"2021-09-29T21:14:09Z"}
Environment:
- Shell-operator version: v1.0.3
- Kubernetes version: 1.20 on eks
- Installation type (kubectl apply, helm chart, etc.):
kubectl apply
Hello! I've never seen Ruby hooks, thank you for these examples!
Q: sometimes it is combined and tasks are dropped from the queue, other times all tasks are executed A: Context combining inconsistency is not a bug. Events from Kubernetes come with unpredictable timings, so the second event can arrive during the hook execution for the first event — there is no combining in this situation. You can see this in logs:
"msg":"kubernetes[0]{OnCreateDeleteArgoApp}: Deleted argocd/Application/aut-410: send KubeEvent","time":"2021-09-29T18:56:57Z"}
"msg":"Create tasks for 'kubernetes' event 'Event 'Deleted' for argocd/Application/aut-410'","time":"2021-09-29T18:56:57Z"}
"msg":"queue task HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp","operator.component":"handleEvents","queue":"main","task.id":"e56af739-fa33-4403-8503-5ba36a6e08c1","time":"2021-09-29T18:56:57Z"}
...
"msg":"Executing command '/hooks/integrations.rb' in '/hooks' dir","queue":"main","task":"HookRun","time":"2021-09-29T18:56:57Z"}
"msg":"kubernetes[0]{OnCreateDeleteArgoApp}: Deleted argocd/Application/aut-369: send KubeEvent","time":"2021-09-29T18:56:57Z"}
...
"msg":"queue task HookRun:main:kubernetes:integrations.rb:OnCreateDeleteArgoApp","operator.component":"handleEvents","queue":"main","task.id":"1546bd1e-c318-4db1-9074-59c204735aef","time":"2021-09-29T18:56:57Z"}
...
"msg":"Executing command '/hooks/integrations.rb' in '/hooks' dir","queue":"main","task":"HookRun","time":"2021-09-29T18:56:58Z"}
Sometimes both events arrive quickly and tasks are queued fast and contexts are combined.
Context combining is just a helper to withstand high loads, it can't be consistent with kubectl commands.
Q: Sometimes it runs for only one resource and sometimes two. A: I see that the context is parsed into array with two objects:
fit-qa-integrations-588c8bfb76-rbntf fit-qa-integrations {"binding":"OnCreateDeleteArgoApp","event":"kubernetes","hook":"integrations.rb","level":"info","msg":"-----2","output":"stdout","queue":"main","task":"HookRun","time":"2021-09-29T18:57:57Z"}
Can you attach the content of BINDING_CONTEXT_PATH
? You can remove "object" content.
Q: I want to run cleanup tasks based each resource that trigged the Deleted event
A:
Reacting to individual Deleted
event is not reliable for cleanup. Please consider using the snapshot mode and compare two sets of objects to detect deleted.
I observe the same issue with combining. I use forked version of this operator https://github.com/confluentinc/streaming-ops/tree/main/images/connect-operator with latest latest version of flant/shell-operator:v1.0.7. And when we create/update/delete more than one ConfigMap hook runs only for one of them. In logs there is a message:
"Binding contexts from 3 tasks are combined to 3 contexts. 2 tasks are dropped from queue 'main'"
How can we force running hook for all ConfigMaps?
@diafour any thoughts how can we force running hook for all ConfigMaps?
I have also noticed this issue since 1.0.8
(currently running 1.0.9
). In the context of resources managed by ArgoCD with self-healing enabled, if the resource is deleted and then created (via ArgoCD self-healing) seconds afterwards, the watchEvent: Added
and even the watchEvent: Modified
are sometimes combined with the Deleted
event.