feature request: better log format and output

[tarrantro]

Describe the feature More clear logs and error handler.

Your proposal

For now, it's very hard to parse jina executor logs, that will cause user setup alerts diffcultly. I'd suggest we can stream logs in below format(better format will be more appriciated) [LOG LEVEL] [Client] [Endpoint] [Module] [MESSAGE] [STATUS_CODE] [Duration] [TIME]

Above request is came from the alert setting in jcloud. We plan to setup executor alerts from logs by searching keyword, however, it's very hard for the team to parse logs because the jina output doesn't have unitfied output format and usefull error message.

---

Environment jcloud + loki + grafana + jina 3.9.1

Screenshots

[JoanFM]

Hey @tarrantro ,

I need to understand:

• What is Client here?
• What is module? I guess is Python module
• Status Code? Not every log has a status code related right? What is the intended work here?
• What is Duration? A log does not have duration itself

[tarrantro]

• What is Client here? I was thinking if we can have the IP address that sending the requests to executor, this is optional.
• What is module? I guess is Python module Yes, that will be better if we can have the .py file to where reports the issue.
• Status Code? Not every log has a status code related right? What is the intended work here? For the gateway or API, we can parse the status code for alerting. That is also optional if hard to do that.
• What is Duration? A log does not have duration itself I mean the process time, can we have that in logs so we can know how much time in each processing?, session or requests.

I think we should have at least base format like: [LOG LEVEL] [Module] [MESSAGE] [TIME]. so we have parse the logs and retrieve keywords from [MESSAGE]. Right now, the logs are scattering, we have lots of new-line so the logs will split and hard to parse. Also, not much useful error message, we can only check on exception which is hard to scrape in log query.

[JoanFM]

For @samsja to review.

My take is:

• Client: I do not think this is feasible, also for other clients we may not have access to this information
• Module: I am sure we can have the module, not sure about file.
• Status code: I am not sure it makes sense to associate a Status code with a log, I think this may be related to Tracing. Please talk to @girishc13 on how u want to implement this alerting, maybe it should be related to his work
• Duration: I also do not see the point here. I think this should be related to Monitoring @samsja

[girishc13]

Tracing is relatively new and might not be heavily used by everyone because its an added cost to maintain a tracing collector or buy a tracing platform for analysis.

Logging on the other hand is always available and the default tool for observing an application. Its fine to have the request status in the log because it allows us to easily filter requests that are problematic (failing, high latency, etc.). Just having the status will not be helpful there must be other identifier information such as client ip, request params, request url, feature flags etc.

Json logging format is also very popular because it can be easily exported over http and stored in any NoSQL type database. Almost all logging libraries and third party log analysis platforms (Scalyr, Datadog, LogStash) support json format. This allows us to create a custom parser and run custom queries to detect anomalies and issues.

[samsja]

Client: I do not think this is feasible, also for other clients we may not have access to this information Module: I am sure we can have the module, not sure about file.

I share some of the concern of @JoanFM. Client and Module can't actually be used.

After talking to @girishc13 I believe than to support what @tarrantro want we should switch to a JSON based logging system (i.e. we log a full json instead of a line). With it we could have just column that are empty when we are not logging about a requests (status code etcc )

Nevertheless I believe that switching to a JSON based logging is a tradeoff between human readability and human debugging and ability to collect log by a third party tool.

IMO we can have a CloudLogger that will be used when the flow is deployed on Jcloud or in k8s and for local we use our classic line logger

[deepankarm]

Rethinking this, we might just need to expose one for logging.jcloud.yml under jina/resources and set JINA_LOG_CONFIG=jcloud for all deployments in JCloud.

Depending on the best practices of alerting from logs, we can use a Handler with JSONFormatter or any other structured formatter. @tarrantro Since JCloud team would know the best practices, we should own this.

[tarrantro]

surely we can only the format handler in Jcloud. However, this issue is about the jina executor logs output is not structed and we or users is very hard to parse it in code. It's human friendly, but not structed in a way for machine to read it.

[tarrantro]

FYI, this is how it looks in loki/grafana. It's very hard to parse for alerts or searching.

[tarrantro]

Another example, nginx logs. I can easily parse logs and set alerts by status code.

[deepankarm]

surely we can only the format handler in Jcloud. However, this issue is about the jina executor logs output is not structed and we or users is very hard to parse it in code. It's human friendly, but not structed in a way for machine to read it.

My point is not about using this logger in JCloud operator/current wolf. It is about

• Add a new logging.jcloud.yml file with JSONFormatter in jina under jina/resources.
• For every K8S deployment, set envvar JINA_LOG_CONFIG=jcloud so that jina uses logging.jcloud.yml for all its logging. So Executors/Gateway will emit logs in the format we have mentioned.

[samsja]

surely we can only the format handler in Jcloud. However, this issue is about the jina executor logs output is not structed and we or users is very hard to parse it in code. It's human friendly, but not structed in a way for machine to read it.

My point is not about using this logger in JCloud operator/current wolf. It is about

* Add a new `logging.jcloud.yml` file with `JSONFormatter` in jina under [jina/resources](https://github.com/jina-ai/jina/tree/master/jina/resources).

* For every K8S deployment, set envvar `JINA_LOG_CONFIG=jcloud` so that jina uses `logging.jcloud.yml` for all its logging. So Executors/Gateway will emit logs in the format we have mentioned.

I completely agree

[tarrantro]

I mean, besides the formatter. We have lots of new-line in current log message, which will scatter the logs output, as well as the exception. @deepankarm , we should handle those as well, or the json format will still have logs of message which can't be parse.