jitsu
jitsu copied to clipboard
Use main events queue in users recognition service
Problem
On every incoming event, Jitsu writes JSON payload to the separate in-memory queue in (server/users/service.go func Event()). It can lead to increased RAM consumption. It is possible to remove separate users recognition queue and do recognition logic in the main event pipeline - after getting event from the main events queue (server/storages/streaming.go func start()) if a destination is in stream mode or after event deserialization from the batch file (server/logfiles/uploader.go) if a destination in batch mode.
Solution
- [ ] Remove
server/users/service.go Event()call from events post processors (server/eventspkg) - [ ] Move (
server/users/servoce.go processRecognitionPayload()func) call after enrichment step. - [ ] Consider that anonymous events will be stored differently according to #724