pamburus
interactive filter
Hi. Awesome work
But I didn't get why user can't change filtering options in interactive mode. Coming from HyperDX land, that's unexpected.
Hi, thanks. Yes, an interactive mode would be very useful. I've been thinking about it lately, I'll probably try to implement something soon. I just wanted to finish one more feature first.
I feel like if we were to support this feature, hl must provides its own terminal viewer based on other rust lib instead of relying on external viewer command tools such as less. On top of that, less is not available on Windows which is a pain to use.
I agree that installing Less on Windows is not as easy as it may seem. There are several ways to get Less on Windows.
- Install jftuga/less-Windows. This link can be found on the official website. I tried it and can say that it is quite buggy, at least visually. It breaks a lot of ANSI escape sequences and everything looks kind of crooked. It also seems to be quite slow.
- Build it from source with Virtual Studio or with gcc for mingw64. I have not tried this yet, it requires too much effort.
- Install it from Scoop with
scoop install less. - Install it from Chocolatey with
choco install less.
The builds from both the Scoop and Chocolatey repositories work perfectly on Windows 11 2024H2 and Windows Terminal. Nothing is broken and the performance is impressive. Added corresponding notes to the README.md for the Windows users about the proper way to install Less on Windows.
Please try this first, and if you still feel that adding TUI to hl is very necessary, and can't wait and you have enough desire and time for it, you can at least try to make a prototype. But be warned, the current project structure is a bit messy. This is my first project in Rust, and I struggled a lot until I got some experience and confidence in Rust. So, I don't think the current state of the project is well suited for active open contributions of such significant size as TUI support. If you can find a way to create a prototype without refactoring existing code, you're welcome. I think using ratatui is probably the best way. But I really doubt that it is possible to add TUI support without significant refactoring of the existing code.
I would like to avoid adding another refactoring at this point, because I am already in the middle of a massive refactoring of the internal data model and parsing. Since I only work on this project when I have some free time and am not overloaded with my main job, I expect this refactoring to be finished in a month or two at best. And I believe that this refactoring will make it much easier to add TUI support. At least it gets rid of annoying lifetime parameters and allows free passing of individual log records. Currently, everything is tied to arena-allocated segments and all records borrow the source data from the segment. However, I have a feeling that it will probably require another refactoring or two. For example, it seems to me that loading data in the opposite or random direction on demand will require additional refactoring. Currently, the pipeline is set up once and then all data is processed according to the sequence originally set up with the pipeline. There is some random access stuff in --sort mode, but I don't think it will fit perfectly in TUI scenarios.
Probably the easiest way to add a minimalist TUI without refactoring existing code, and that behaves similarly to Less, is to just make it work at the binary stream level over some pipes or channels. But I don't see much point in that, except that it allows you to use the built-in pager without any additional binary dependencies. Installing Less is not that difficult. If you want to make a generic pager that is an alternative to Less, you can do it as a separate project, but this will involve a lot of pain in handling ANSI escape sequences and other control characters.
The real value of adding TUI is the ability to conveniently navigate between log records, change filters on the fly, toggle other display options such as sorting, or hide/show some fields that clutter the screen, focus on some fields of interest, going back to the previous view, and so on. That's not an easy task, but I think that's where this project needs to go.
Hello, thanks for hl it's amazing!
Reading this thread I thought I could share how I'm using hl at work to debug a distributed system.
The gist is to describe a "case" in a yaml file (chose this format for its brevity), which then drives a little script that uses hl to parse logs.
First there's a list of conditions that characterize the bug, because the system is not fully deterministic and I want to be able to regenerate the bug logs after changing logs & tracing. Here's for example a bug case I'm working on in which one client syncs and the other does not:
test: two_clients_converge_with_server_after_drawing_lines_with_timeouts
seed: 0x72345eee00000020
bug_conditions:
- matching_lines: 1
filters:
- message=Not synced
- client_id=client_0
- target=rayon_sync_fuzzer::client
- matching_lines: 1
filters:
- message=Synced
- client_id=client_1
- target=rayon_sync_fuzzer::client
The script runs the given test+seed in a loop until all conditions match, and the log file is saved for later examination.
Then in the same file I define different "angles" to look at the logs after the bug has been reproduced:
default_angle: raw_log
angles:
raw_log: {}
not_synced_client:
query: 'target~~=rayon_sync_client::.* or target~~=rayon_sync_fuzzer::.*'
filters:
- client_id=client_0
synced_client:
query: 'target~~=rayon_sync_client::.* or target~~=rayon_sync_fuzzer::.*'
filters:
- client_id=client_1
server:
filters:
- client_id~=client_
- 'log.target~=rayon_sync::'
hide:
- '*'
all_peers:
query: 'target~~=rayon_sync_client::.* or target~~=rayon_sync_fuzzer::.* or log.target~~=rayon_sync::.*'
filters:
- client_id~=client_
hide:
- '*'
- '!client_id'
- '!since_version'
- '!target'
- '!target-log'
- '!span-path'
- '!last_pulled_version'
- '!push_queue'
This allows me to easily switch views on the same logs and put the pieces together.
I wanted to share this because I considered writing a TUI at first, and then I thought about using a conf file, which I can commit to the repo, come back to later, etc...
Hope that fuels some thoughts, thanks again for the amazing tool!
