jitsu icon indicating copy to clipboard operation
jitsu copied to clipboard

Use main events queue in users recognition service

Open xtreding opened this issue 4 years ago • 0 comments

Problem

On every incoming event, Jitsu writes JSON payload to the separate in-memory queue in (server/users/service.go func Event()). It can lead to increased RAM consumption. It is possible to remove separate users recognition queue and do recognition logic in the main event pipeline - after getting event from the main events queue (server/storages/streaming.go func start()) if a destination is in stream mode or after event deserialization from the batch file (server/logfiles/uploader.go) if a destination in batch mode.

Solution

  • [ ] Remove server/users/service.go Event() call from events post processors (server/events pkg)
  • [ ] Move (server/users/servoce.go processRecognitionPayload() func) call after enrichment step.
  • [ ] Consider that anonymous events will be stored differently according to #724

xtreding avatar Dec 16 '21 08:12 xtreding