fix(core/worker_events): overwrite the `post` method of worker_events
Summary
There is a limit for the size of payload allowed to be send in events lib. In some cases, Kong needs to send big payload. For example DAO:crud rely on events to invalidate cache. In these cases, the size of the payload may exceeds the limit due a large dao entity.
In this PR, the post method of the worker_events is overwritten to take a retry action when we get the 'payload too big' error message, and the retry action is just to re-post the event with a truncated payload so that the subsequent logic that relies on the event could have a chance to be informed rather than unaware at all. Based on this purpose, the retry action is taken and only taken once when the error happens.
The limit for the size of payload is defined inside the overwritten post method as a conventional value.
Checklist
- [x] The Pull Request has tests
- [x] There's an entry in the CHANGELOG
- [ ] There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE
Full changelog
- [Implement ...]
Issue reference
FTI-4963 KAG-1523
This one is waiting for a co-related PR https://github.com/Kong/lua-resty-events/pull/28 to be merged and bumped
lua-resty-events has bumped to 0.1.5. See: https://github.com/Kong/kong/pull/10883
Could we put the tests spec/02-integration/01-worker_events_spec.lua into a proper directory?
In a real world, where do we have this issue happening?
The description says:
DAO:crud rely on events to invalidate cache. In these cases, the size of the payload may exceeds the limit due a large dao entity.
But where exactly, and why DAO crud event can be so big? Should we make it a bit more intelligent? E.g. truncate the fields or something? Or fix it at the source of event?
In a real world, where do we have this issue happening?
For some plugin objects whose fields are not size-limited may be too large. As FTI-4963 suggested, the config field of request-validator plugin is too large for worker_event lib to send events. So large events are basically caused by size-unlimited fields.
Should we make it a bit more intelligent? E.g. truncate the fields or something?
I'm afraid it would be a bit flaky to try to truncate large fields of an entity, fields to be truncated might vary with the specific entities.
Or fix it at the source of event?
Fixing it at the source of event means we have to limit the size of fields that are unlimited initially, this would introduce a breaking change which might not such necessary at this moment.
This PR could be closed as we had another approach for this, referring to https://github.com/Kong/lua-resty-events/pull/37