Logging session history as structured data

[monkey-w1n5t0n]

Related problem

I am trying to keep automatic daily logs of all my activity while working, including terminal sessions. So far I haven't found a way to export my Nushell sessions as structured data so that I can save, reopen, scroll through, and programmatically analyze them.

Describe the solution you'd like

It would be amazing if Nushell could be configured to serialize and automatically log user sessions in a structured data format, so that we could then:

1. Reopen them in Nushell at a later date and scroll through - as if we had never closed the terminal window at all.
2. Read and process the files programmatically to extract interesting and useful metrics/statistics, or for troubleshooting purposes.

With Nushell's powers for processing structured data, this could result in some really cool scripting abilities. For example:

• "Display all commands that I ran in directory X last week that failed with exit code 1"
• "When was the last time that running cargo build in directory Y succeeded?"
• "Show me all directories I have ever executed a git commit in and sort by commit frequency/total number/average number of lines added/etc."
• "Find all times I ran cat on somefile.txt (from any directory) this year and save each output as somefile_history/$date.txt"

This is just what I can come up with off the top of my head - I'm sure others will eventually come up with much more interesting and useful ideas if this ends up being implemented.

As part of the configuration, it could be possible to set some kind of limit for how long of a history to keep (either size-based, time-based, number of commands etc), exclude certain commands from being logged (either entirely or just hiding the output), or add some kind of hook so that whenever command X is executed and logged, command Y is also executed and its output is attached to command X's log (e.g. automatically logging the output of ls in a directory whenever executing a command/script that might add or delete files without explicitly notifying the user).

Describe alternatives you've considered

I tried using the script utility to record terminal sessions, but the output is practically unreadable (not properly formatted) and it's not really structured data.

I tried to see whether Alacritty has the ability to serialize and restore user sessions, but it doesn't seem to. Even then, it still wouldn't be structured data, so it would mostly be useful for manually scrolling through the history.

Additional context and details

No response

[sholderbach]

We have the option to use a history with sqlite based storage that keeps additional metadata compared to the basic text file that only llists the commands.

let-env config = {
  ...
  history_file_format: "sqlite" # "sqlite" or "plaintext"
  ...
}

In theory you should be able to do your personal analysis on this file with the provided database/sqlite commands. But the integration to directly use it interactively in the shell has not been implemented yet.

If you are interested in helping out in this area this would be appreciated: For example we currently don't support a way to suppress storing certain entries (other shells handle that with spaces in front of the code, but potentially having patterns to detect secret tokens would also be possible)

[monkey-w1n5t0n]

Thanks for the reply - I must have missed the history_file_format in the config file!

But the integration to directly use it interactively in the shell has not been implemented yet.

Correct me if I'm wrong, but wouldn't that come for free if the data was stored in a structured format that nu can already work with (e.g. JSON), instead of the opaque storage of an SQLite database?

If you are interested in helping out in this area this would be appreciated:

Happy to contribute as best I can, but I'm new to the codebase and not all that experienced with Rust so it may take some time to get up to speed.

To begin with, I assume that on the implementation side there should be a function that takes in a given command and returns a boolean of whether it should be logged or not. To be thorough, that function should not just consider the raw text that was typed by the user, but should take into account a number of other bits of information, such as:

• The current directory - e.g. the user might be happy to log all outputs of the cat command, except for any that may occur in ~/ultra_secret_passwords_stored_in_plaintext
• Any alias expansions
• The current ENV, to be able to expand things like paths
• more?

Would such a function belong in the into_sqlite command, or somewhere further up the chain?

Then there's also the question of how the user should specify what should get logged and what not, for anything more complex than "if the command contains the string XYZ". Perhaps by providing their own predicate in the form of a function that takes in the above state and returns a bool?