lnav
lnav copied to clipboard
tstack
Using Arbitrary Timestamp Formats
I have a situation where my time-stamps don't actually represent a unix-time, but rather the number of picoseconds into a simulation. I can't figure out if there is away to define a log format where the timestamp is just a field that matches the regex "(\d+) ps".
Ive tried defining timestamp fields using the %i, %Y ... modifiers, but it looks like they expect an exact number of digits.
Would it be possible to add this feature, or is the use of unix time formats hardwired somewhere deep?
I think "%i" should work for you, it shouldn't look for a fixed number of digits. However, it treats the value as milliseconds from the epoch and not pico seconds. Also, when displaying the log message, lnav will automatically convert the numeric epoch timestamp into a human readable date, which you may not want.
Your use case is valid, but not something that's handled at the moment.
Sorry for the late reply.
I am facing a problem while trying to define a new format when I use a log line with a custom timestamp as shown in the following example:
01 Nov 2017 00:00:01,666 [INFO] bla bla bla
For this I define the following pattern in my .json file:
^(?<timestamp>\\d{1,2}\\s+\\w{3}\\s+\\d{4}\\s+\\d{2}:\\d{2}:\\d{2},\\d{3})\\s+\\[(?<alert_level>\\w+)\\]\\s+(?<body>.*)$
But then I receive an error saying that:
error:log4j_ex:unrecognized timestamp format -- 01 Nov 2017 00:00:01,666 [INFO] bla bla bla
and throws a list of acceptable timestamp formats:
format: @%@; matched:
format: %Y-%m-%d %H:%M:%S; matched: 01 N
format: %Y-%m-%d %H:%M:%S%z; matched: 01 N
format: %Y-%m-%d %H:%M:%S %z; matched: 01 N
format: %Y-%m-%d %H:%M; matched: 01 N
format: %Y-%m-%dT%H:%M:%S.%f%z; matched: 01 N
format: %y-%m-%dT%H:%M:%S.%f%z; matched: 01
format: %Y-%m-%dT%H:%M:%SZ; matched: 01 N
format: %Y-%m-%dT%H:%M:%S%z; matched: 01 N
format: %Y-%m-%dT%H:%M:%S; matched: 01 N
format: %Y-%m-%dT%H:%M:%S%z; matched: 01 N
format: %Y/%m/%d %H:%M:%S; matched: 01 N
format: %Y/%m/%d %H:%M:%S %z; matched: 01 N
format: %Y/%m/%d %H:%M:%S%z; matched: 01 N
format: %Y/%m/%d %H:%M; matched: 01 N
format: %a %b %d %H:%M:%S %Y; matched: 01 Nov 20
format: %a %b %d %H:%M:%S.%f %Y; matched: 01 Nov 20
format: %a %b %d %H:%M:%S %Z %Y; matched: 01 Nov 20
format: %a %b %d %H:%M:%S ; matched: 01 Nov 20
format: %a %b %d %H:%M:%S.%L ; matched: 01 Nov 20
format: %d/%b/%Y:%H:%M:%S +0000; matched: 01
format: %d/%b/%Y:%H:%M:%S %z; matched: 01
format: %b %d %H:%M:%S; matched:
format: %b %d %k:%M:%S; matched:
format: %b %d %l:%M:%S; matched:
format: %b %e, %Y %l:%M:%S %p; matched:
format: %m/%d/%y %H:%M:%S; matched: 01
format: %m/%d/%Y %I:%M:%S:%L %p %Z; matched: 01
format: %m/%e/%Y %l:%M:%S %p; matched: 01
format: %m%d %H:%M:%S; matched: 01
format: %H:%M:%S; matched: 01
format: %M:%S; matched: 01
format: %m/%d %H:%M:%S; matched: 01
format: %Y-%m-%d; matched: 01 N
format: %Y-%m; matched: 01 N
format: %Y/%m/%d; matched: 01 N
format: %Y/%m; matched: 01 N
Does this mean that I cannot use my custom timestamp? Thank you!
Sorry to bump an old issue, but I have a similar problem to solve. My text format timestamps are of the form:
12345.9
They represent the fractional number of milliseconds passed since the start of log recording. An absolute start time offset is listed in some header line at the top of each file, but that's not really important.
I've tried some timestamp-format definitions, but none worked so far:
"%L.%f"
"%i.%f"
It seems that %f always wants six digits. Dropping the part after the decimal point would lose needed precision, the best solution would be to interpret the string as a floating point number (rounded to one decimal place) and scale it down to milliseconds, as is possible in JSON formats.
I succeeded to read a german / european locale / native formatted date string Aug 3, 2021 10:07:49,502 AM CEST using the following custom / arbitrary timestamp-format: "timestamp-format" : [ "%b %e, %Y %I:%M:%S,%L %p %Z" ],
edit: correction this does not work as neither %d (zero padded day of the month) nor %e (space padded day of the month) will match the non-padded day in the example string. see below comment for some more details.
@salvadom your timestamp 01 Nov 2017 00:00:01,666 should be matching the following
"timestamp-format" : [ "%d %e %Y %H:%M:%S,%L" ],
@acolomb I would try to parse the timestamp somehow excluding the fractional dot in your representation and/or adding the missing trailing zeros during the regex parsing step.
@DanChianucci the current lnav timestamp-format supports only the following:
%LMilliseconds as a decimal number (range 000 to 999).%fMicroseconds as a decimal number (range 000000 to 999999).%NNanoseconds as a decimal number (range 000000000 to 999999999).
I guess it would require changes to the general libstdc++ Time or std:chrono implementation used by most c++ programs as a library for Rational Arithmetic std::pico precision (AFAIR this only supports nanosecond precision at the moment) and then extending the parser in src/base/date_time_scanner.cc and src/ptimec.c.
As the above libstdc++ Time Utilities library states: Highest-resolution clock. This is the clock "with the shortest tick period." Alias to std::system_clock until higher-than-nanosecond definitions become feasible. Definition at line 1163 of file std/chrono.
There is a suggestion that implementation of picosecond precision using Time or chrono is feasible on StackOverflow How to accurately measure time in c++?
@tstack however I could not get the following working:
date string 8-3-2021 00:01:28.510
matching "timestamp-format" : [ "%-d-%-m-%Z %H:%M:%M.%L" },
Apparently the src/ptimec.c Parser does not know how to handle non-zero padded Day and Month values, ie. using a dash between % and the date field:
%dzero padded (01..31) works%-dun-padded (1..31) not supported%_dblank-padded (1..31) not supported Though the general Parser and lnav works fine, this would require preprocessing of timestamps in logfiles to zero-pad them first.
Glibc provides some extensions for conversion specifications. (These extensions are not specified in POSIX.1-2001, but a few other systems provide similar features.) Between the % character and the conversion specifier character, an optional flag and field width may be specified. (These precede the E or O modifiers, if present.)
The following flag characters are permitted:
_(underscore) Pad a numeric result string with spaces.-(dash) Do not pad a numeric result string.0Pad a numeric result string with zeros even if the conversion specifier character uses space-padding by default.^Convert alphabetic characters in result string to upper case.#Swap the case of the result string. (This flag only works with certain conversion specifier characters, and of these, it is only really useful with %Z.)
Many thanks for providing and maintaining lnav !