Custom JSON log parsing error

[MikyStar]

lnav version v0.10.1 is the latest

Describe the bug I'm trying to parse custom JSON logs that would look like something like this :

{"name": "Hello there", "level": "info", "timestamp": "14/04/2022 00:51:30"}
{"name": "Something new", "level": "system", "timestamp": "14/04/2022 00:51:35"}

with a format config file that looks like this :

{
	"$schema": "https://lnav.org/schemas/format-v1.schema.json",
	"custom_log":
	{
		"title": "My JSON logs",
		"file-pattern": ".log",
		"json": true,
		"multiline": false,
		"timestamp-field": "timestamp",
		"timestamp-format": [ "%d/%m/%Y %H:%M:%S" ],
		"timestamp-divisor": 1000,
		"convert-to-local-time": true,
		"hide-extra": false,
		"level":
		{
			"debug"	: "system|info|debug",
			"warning": "warning",
			"error"	: "error"
		},
		"body-field": "name",
		"value":
		{
			"name": { "kind": "string" },
                         "level": { "kind": "string" },
			"timestamp": { "kind": "string" },
			"details": { "kind": "json" },
			"requestID": { "kind": "string", "identifier": true },
			"client": { "kind": "json" }
		},
		"line-format":
		[
			{ "field": "timestamp" },
			" ",
			{
				"field": "level",
				"min-width": 6
			},
			{ "field": "requestID" },
			{ "field": "details" }
		]
	}
}

And if I run :

lnav -i my-format.json my-logs.log

I get :

info: installed: /Users/me/.config/lnav/formats/installed/custom_log.json
warning:my-logs.log:line 1
warning:  unexpected path --
warning:    /name
warning:  accepted paths --
warning:    $schema The URI of the schema for this file -- Specifies the type of this file
warning:    (\w+)  -- The definition of a log file format.
warning:temp.back.log:line 1
warning:  unexpected path --
warning:    /level
warning:  accepted paths --
warning:    $schema The URI of the schema for this file -- Specifies the type of this file
warning:    (\w+)  -- The definition of a log file format.
warning:temp.back.log:line 1
warning:  unexpected path --
warning:    /timestamp
warning:  accepted paths --
warning:    $schema The URI of the schema for this file -- Specifies the type of this file
warning:    (\w+)  -- The definition of a log file format.
error: format file is empty: my-logs.log

Do you have an idea ?

[tstack]

The lnav -i is just for installing format files and log files should not be passed in. Just run

lnav -i myformat-json

And then run:

lnav my-logs.log

to view the file.

[MikyStar]

Hey !

I just tried it, I can indeed enter lnav for my log file but still, the format file is not applied, lnav sees it as plain text and is displaying the raw JSON unbrowsably

If I use the debug option, I can see that lnav is loading the proper format file but seems to have problem regex wise, here's a portion of the stack trace :

....
2022-04-25T12:55:27.739 I t0 log_format_loader.cc:897 loading formats from path: /Users/me/.config/lnav/formats/*/*.json
2022-04-25T12:55:27.739 I t0 log_format_loader.cc:840 loading formats from file: /Users/me/.config/lnav/formats/installed/custom_log.json
2022-04-25T12:55:27.739 D t0 log_format_loader.cc:86 Loading format -- custom_log
2022-04-25T12:55:27.739 I t0 log_format_loader.cc:917   found format: custom_log
2022-04-25T12:55:27.739 W t0 log_format.cc:1647 alb_log/regex/std:body field 'body' not found in pattern
2022-04-25T12:55:27.739 W t0 log_format.cc:1636 block_log/regex/sq-brackets:level field 'level' not found in pattern
2022-04-25T12:55:27.739 W t0 log_format.cc:1636 block_log/regex/std:level field 'level' not found in pattern
2022-04-25T12:55:27.740 W t0 log_format.cc:1636 candlepin_log/regex/other:level field 'level' not found in pattern
2022-04-25T12:55:27.740 W t0 log_format.cc:1636 candlepin_log/regex/reqorg:level field 'level' not found in pattern
2022-04-25T12:55:27.740 W t0 log_format_loader.cc:985 Format collision, format 'cups_log' matches sample from 'error_log'
2022-04-25T12:55:27.740 W t0 log_format.cc:1636 dpkg_log/regex/std:level field 'level' not found in pattern
2022-04-25T12:55:27.740 W t0 log_format_loader.cc:985 Format collision, format 'error_log' matches sample from 'cups_log'
2022-04-25T12:55:27.740 W t0 log_format.cc:1636 fsck_hfs_log/regex/std:level field 'level' not found in pattern
2022-04-25T12:55:27.740 W t0 log_format.cc:1636 haproxy_log/regex/event_started:level field 'level' not found in pattern
2022-04-25T12:55:27.740 W t0 log_format.cc:1647 haproxy_log/regex/event_started:body field 'body' not found in pattern
2022-04-25T12:55:27.740 W t0 log_format.cc:1636 haproxy_log/regex/event_stopped:level field 'level' not found in pattern
2022-04-25T12:55:27.740 W t0 log_format.cc:1647 haproxy_log/regex/event_stopped:body field 'body' not found in pattern
2022-04-25T12:55:27.740 W t0 log_format.cc:1636 haproxy_log/regex/event_stopping:level field 'level' not found in pattern
2022-04-25T12:55:27.740 W t0 log_format.cc:1647 haproxy_log/regex/event_stopping:body field 'body' not found in pattern
2022-04-25T12:55:27.740 W t0 log_format.cc:1636 haproxy_log/regex/http:level field 'level' not found in pattern
2022-04-25T12:55:27.740 W t0 log_format.cc:1647 haproxy_log/regex/http:body field 'body' not found in pattern
2022-04-25T12:55:27.740 W t0 log_format.cc:1636 haproxy_log/regex/ssl:level field 'level' not found in pattern
2022-04-25T12:55:27.740 W t0 log_format.cc:1647 haproxy_log/regex/ssl:body field 'body' not found in pattern
2022-04-25T12:55:27.740 W t0 log_format.cc:1636 haproxy_log/regex/tcp:level field 'level' not found in pattern
2022-04-25T12:55:27.740 W t0 log_format.cc:1647 haproxy_log/regex/tcp:body field 'body' not found in pattern
2022-04-25T12:55:27.743 W t0 log_format.cc:1647 katello_log/regex/log:body field 'body' not found in pattern
2022-04-25T12:55:27.744 W t0 log_format.cc:1636 openamdb_log/regex/std:level field 'level' not found in pattern
2022-04-25T12:55:27.744 W t0 log_format_loader.cc:985 Format collision, format 'openstack_log' matches sample from 's3_log'
2022-04-25T12:55:27.744 W t0 log_format.cc:1636 page_log/regex/1.7:level field 'level' not found in pattern
....

[tstack]

The file-pattern is a regular expression that needs to match the log file name. Try changing it to something like .*\.log so that it will match file names ending with .log.

The detection logic for JSON file isn't very good at the moment, it only relies on this file name pattern to figure out which format to use. It's something that needs to be improved.

[MikyStar]

I tried a few things yet I got the same behaviour

But from the debug logs above, it looks like lnav is loading the proper format file, it just doesn't seems to stick with it

[tstack]

I changed the file-pattern property to look like the following and it seems to work fine:

"file-pattern": ".*\\.log",

Here's a screenshot:

[MikyStar]

Wow, well I don't know what to say, I just tried it again and it's not doing it for me

Are you running 0.10.1 ? I installed it from homebrew

[tstack]

Did you change the format file in "~/.lnav/formats/installed/custom_log.json"? If not, you'll need to rerun lnav -i to install the updated version.

[MikyStar]

Yes I updated it