move log level inside logging system

[532910]

% journalctl -u neofs-storage.service -p err      
-- No entries --

The No entries is wrong answer, because there are:

% journalctl -u neofs-storage.service | grep error | wc -l
123998

[roman-khimov]

That's called severity. I thought it works fine, but if we're catching these lines from stdout then journald docs should be checked for possibility of severity detection from them.

[532910]

That's called severity

in syslog

in systemd it's priority: https://manpages.debian.org/bookworm/systemd/journalctl.1.en.html

-p, --priority= Filter output by message priorities or priority ranges. Takes either a single numeric or textual log level (i.e. between 0/"emerg" and 7/"debug"), or a range of numeric/text log levels in the form FROM..TO. The log levels are the usual syslog log levels as documented in syslog(3), i.e. "emerg" (0), "alert" (1), "crit" (2), "err" (3), "warning" (4), "notice" (5), "info" (6), "debug" (7). If a single log level is specified, all messages with this log level or a lower (hence more important) log level are shown. If a range is specified, all messages within the range are shown, including both the start and the end value of the range. This will add "PRIORITY=" matches for the specified priorities.

[roman-khimov]

I doubt zap can do prefixes like in https://www.freedesktop.org/software/systemd/man/latest/sd-daemon.html. It also doesn't have a proper set of severities/priorities/levels.

[532910]

https://git.frostfs.info/TrueCloudLab/zapjournald it was the first link by google(go zap journald)

[532910]

also journald is supported by zap: https://pkg.go.dev/go.pact.im/x/zapjournal

[roman-khimov]

https://git.frostfs.info/TrueCloudLab/zapjournald

That one solves a different problem, but maybe can be extended to handle levels in some way.

https://pkg.go.dev/go.pact.im/x/zapjournal

Also different.

There are two problems here:

• zap levels are not syslog severities https://datatracker.ietf.org/doc/html/rfc5424#section-6.2.1
• it needs to translate and output them to syslog in some way (sd-daemon is one of the simplest there, still stdout, but something journald can parse)

[fyfyrchik]

https://git.frostfs.info/TrueCloudLab/zapjournald

We have tried it (actually, have written), so I will just outline some problems we have encountered:

1. Zap is famous for being zero-alloc and fast, it was harder to achieve this with zapjournald (we still have 1 alloc for trivial cases, but this is because of our requirement to have other native journal fields without them being used as application-level fields). In neo-go, it could be made zero-alloc.
2. We have reverted the change in one of our services, because custom journal sink for zap had random slowdowns (likely related to journald itself, still investigating).

[roman-khimov]

I don't see any solution to the original problem, however I should point that JSON logs added in #3838 can be a replacement in many cases.