haystack
haystack copied to clipboard
Add target persona meta information to logs
As a datascientist I don't want my logs to be cluttered with devops infos.
Is your feature request related to a problem? Please describe.
Currently we do not separate logs according to the persona. devops infos (e.g. availability of multiprocessing
, numba
, number of worker processes, telemetry) dilutes the logs a data scientist is interested in. Same applies for devops logs regarding data scientist infos (e.g. warnings about which similarity metric to use for some models).
Describe the solution you'd like
We add extra haystack_persona
to all logs:
- value
datascience
marks logs for data scientists - value
devops
marks logs for devops/data engineers - Multiple values are allowed. The type of
haystack_persona
is an array of values.
Describe alternatives you've considered Another distinction could be (user-facing vs. backend-applications) but although this has almost the same distribution it is way harder to grasp for notebook users.
Additional context
Main focus lies on bigger deployments using analytics and monitoring tools and a separate UI. Notebook users should also be able to make use of it. The value must be easily filtered for in standard logging
and structlog
.