Improved structured logging

[dveeden]

Feature Request

Starting TiDB with the log format set to JSON allows for structured logging.

./bin/tidb-server -config <(echo -en '[log]\nformat="json"')

This could be used to expand fields that are now all combined together into the message column of ES.

$ ./bin/tidb-server -config <(echo -en '[log]\nformat="json"\n[log.file]\nfilename="/tmp/tidb.log\n') &
$ jq 'select(.caller == "server.go:248")' /tmp/tidb.log
{
  "level": "INFO",
  "time": "2022/01/13 13:26:25.415 +01:00",
  "caller": "server.go:248",
  "message": "server is running MySQL protocol",
  "addr": "0.0.0.0:4000"
}

[dveeden]

Using structured logging actually causes the scraper to give up. The result is that the file is not collected at all and isn't part of the the set of files that is uploaded to the clinic server, etc.

[dvaneeden@centos70-ne ~]$ /tmp/tiup/bin/scraper --log '/tidb-deploy/tidb-4000/log/*' -f'-48h' -t='-0h'
error parsing tidb.log: the first line is not a valid log line
{
  "log_files": {
    "/tidb-deploy/tidb-4000/log/tidb_stderr.log": 0,
    "/tidb-deploy/tidb-4000/log/tidb_slow_query.log": 5007811
  }
}
[dvaneeden@centos70-ne ~]$ head -1 /tidb-deploy/tidb-4000/log/tidb.log | jq .
{
  "level": "INFO",
  "time": "2022/01/14 08:09:55.307 +01:00",
  "caller": "printer.go:34",
  "message": "Welcome to TiDB.",
  "Release Version": "v5.3.0",
  "Edition": "Community",
  "Git Commit Hash": "4a1b2e9fe5b5afb1068c56de47adb07098d768d6",
  "Git Branch": "heads/refs/tags/v5.3.0",
  "UTC Build Time": "2021-11-24 13:32:39",
  "GoVersion": "go1.16.4",
  "Race Enabled": false,
  "Check Table Before Drop": false,
  "TiKV Min Version": "v3.0.0-60965b006877ca7234adaced7890d7b029ed1306"
}
[dvaneeden@centos70-ne ~]$ head -1 /tidb-deploy/tidb-4000/log/tidb.log 
{"level":"INFO","time":"2022/01/14 08:09:55.307 +01:00","caller":"printer.go:34","message":"Welcome to TiDB.","Release Version":"v5.3.0","Edition":"Community","Git Commit Hash":"4a1b2e9fe5b5afb1068c56de47adb07098d768d6","Git Branch":"heads/refs/tags/v5.3.0","UTC Build Time":"2021-11-24 13:32:39","GoVersion":"go1.16.4","Race Enabled":false,"Check Table Before Drop":false,"TiKV Min Version":"v3.0.0-60965b006877ca7234adaced7890d7b029ed1306"}

[AstroProfundis]

cc @qiffang FYI