History file moves to `$nu.data-dir`

[tombh]

Fixes #10100

Consensus was that the history file was not config and so should not live in the config directory by default.

Although #10100 came to agree on using $XDG_STATE_HOME, I ended up implementing $XDG_DATA_HOME simply because the dirs crate that Nushell uses to get default values for these paths doesn't have entries for $XDG_STATE_HOME on OSX and Windows. See: https://docs.rs/dirs/5.0.1/dirs/fn.state_dir.html

Also includes an automated "migration" that moves the old history file to the new path: only if the old file exists and there is no file in the new path.

Notes:

• Changes nu_path::data_dir() to nu_path::nu_data_dir() so that it returns the data dir with /nushell appended. This was already being done by callers of nu_path::data_dir() so I just refactored.

User-Facing Changes

The old history file will automatically be moved to the new path: $XDG_DATA_HOME/nushell/history.{txt,sqlite}

Tests

• history_import.rs tests now set XDG_CONFIG_HOME and XDG_DATA_HOME, there may be other tests that will come to need both. But currently all tests pass.

[fdncred]

This is starting to get a little confusing. What I was agreeing to was that the history file should be in dirs::config_dir() by default unless XDG_CONFIG_HOME was set, maintaining our exist configuration functionality. However, if XDG_STATE_HOME was set, the history file would be placed there.

[NotTheDr01ds]

@fdncred I might be reading the code wrong, but if my recollection of canonicalize_path() is right, I believe it will work the way you describe. Will have to test to be sure.

I definitely agree with your description of the way we want it to work.

[NotTheDr01ds]

However (and I also might be reading the code wrong in this case), it appears that this will likely generate an error on first launch when there is no history file to rename?

I'm also worried about cases where there is an existing 0.100 Nushell that has the history file open, and then the user launches an updated version with this change - Renaming it out from under the older, still running Nushell's might be problematic?

[fdncred]

Renaming it out from under the older, still running Nushell's might be problematic?

ya, i'm pretty sure windows could give a sharing violation error.

[tombh]

I'm a bit confused by: "the history file should be in dirs::config_dir() by default unless XDG_CONFIG_HOME was set". dirs::config_dir() is only ever called by nu_path::nu_data_dir() which is defined as:

pub fn nu_config_dir() -> Option<AbsolutePathBuf> {
    configurable_dir_path("XDG_CONFIG_HOME", dirs::config_dir).map(|mut path| {
        path.push("nushell");
        path
    })
}

So XDG_CONFIG_HOME already takes precedence and dirs::config_dir() is fallback.

But my reading of #10100 is that the history file should never be placed in the config dir, in any fallback scenario. I don't see any comment suggesting that.

However (and I also might be reading the code wrong in this case), it appears that this will likely generate an error on first launch when there is no history file to rename?

The rename is only attempted if the old history file exists and the new history file doesn't exist.

ya, i'm pretty sure windows could give a sharing violation error.

Good point. What about trying to just copy the history file if there's an error. And then if there's another error just giving up and using the new history file?

[NotTheDr01ds]

I'm not quite sure I want the rename/copy to take place at all, but I'm still digesting that part.

If we do anything, I think it should be a copy-only, rather than a rename. Then let the user know that the new one is being used and that they can delete the old one. @fdncred Your thoughts on the auto-migration?

[fdncred]

Right, the history file is in dir::config_dir unless XDG_CONFIG_HOME is set. That's what configurable_dir_path is doing right now in the main branch. I think this PR changes that to dirs::data_dir unless XDG_DATA_HOME is set. Just trying to make sure I'm reading this right.

What about trying to just copy the history file if there's an error.

If the history file is a sqlite db, there is more than one file (.sqlite3, .sqlite3-shm, .sqlite3-wal). This sounds like a recipe for corrupted history unless nushell is exited. We ran into this same problem when trying to come up with a history clear command that just deleted all the files. It was a mess. That --clear flag is partially broken still.

We need more @nushell/core-team opinions on what to do here.

[NotTheDr01ds]

And yes, I see the disconnect now on whether or not there's a scenario in which the history file stays in the config directory. That's going to be tough, I agree. We already have a data_dir defined that is controlled by (if present) XDG_DATA_HOME.

I'm not sure how we could come up with a logical way to keep the history file in $nu.default-config-dir when we already have $nu.data-dir.

Yes, if we had XDG_STATE_HOME, then I guess that would work, but I was never a fan of that one-off in the first place.

Is it correct to say that the real title of this PR should be "History file moves to $nu.data-dir"? The XDG_DATA_HOME isn't really a change here, other than some "clean-up" around it to make it more "generic", right?

[tombh]

The migration aspect of this may turn out to be the hardest part. But that's expected and understandable, let's see what more of the core team thinks.

And, that's a much better title! Changed.

[fdncred]

My history file is already in $nu.data-dir.

❯ $nu
╭──────────────────────┬────────────────────────────────────────────────────────────────────╮
│ default-config-dir   │ /Users/fdncred/Library/Application Support/nushell                 │
│ config-path          │ /Users/fdncred/Library/Application Support/nushell/config.nu       │
│ env-path             │ /Users/fdncred/Library/Application Support/nushell/env.nu          │
│ history-path         │ /Users/fdncred/Library/Application Support/nushell/history.sqlite3 │
│ loginshell-path      │ /Users/fdncred/Library/Application Support/nushell/login.nu        │
│ plugin-path          │ /Users/fdncred/Library/Application Support/nushell/plugin.msgpackz │
│ home-path            │ /Users/fdncred                                                     │
│ data-dir             │ /Users/fdncred/Library/Application Support/nushell                 │
│ cache-dir            │ /Users/fdncred/Library/Caches/nushell                              │
│ vendor-autoload-dirs │ [list 5 items]                                                     │
│ temp-path            │ /private/var/folders/m1/x0gp9jy51s146ttxtz7jlpxh0000gn/T           │
│ pid                  │ 74316                                                              │
│ os-info              │ {record 4 fields}                                                  │
│ startup-time         │ 67ms 594µs 584ns                                                   │
│ is-interactive       │ true                                                               │
│ is-login             │ false                                                              │
│ history-enabled      │ true                                                               │
│ current-exe          │ /Users/fdncred/.cargo/bin/nu                                       │
╰──────────────────────┴────────────────────────────────────────────────────────────────────╯

[NotTheDr01ds]

@fdncred Isn't that only because your $nu.data-dir is the same as your $nu.default-config-dir? If they were different, I think you'll find that the history file is really in $nu.default-config-dir today (pre-PR). This PR would change it to follow the actual $nu.data-dir setting if it was different.

[tombh]

I think that's right and I believe it's because dirs (which we use to provide all these standard paths), doesn't differentiate between config and data dirs on OSX.

[stormasm]

To quote @fdncred from above note...

This is starting to get a little confusing

Here is my setup which seems to be "just fine" with

export XDG_CONFIG_HOME=/Users/ma/.config

Then it yields this simple configuration setup...

default-config-dir   │ /Users/ma/.config/nushell                                             
config-path          │ /Users/ma/.config/nushell/config.nu                                   
env-path             │ /Users/ma/.config/nushell/env.nu                                      
history-path         │ /Users/ma/.config/nushell/history.txt

This reminds of an important saying (in English)...

If it ain't broke don't fix it

I think our solution is pretty good and anything else we do here will just probably make it more confusing (in the code) than it needs to be...

So unless I hear a really good reason to "fix it" my vote would be to leave it alone (for now)...

[amtoine]

i haven't had a look at the implementation for this but, my personal opinion is that having the following files in ~/.local/share/nushell/ would be cleaner than in ~/.config/nushell/ as right now (my $env.XDG_DATA_HOME | path join nushell and $env.XDG_CONFIG_HOME | path join nushell respectively)

• history.txt
• history.sqlite3-shm
• history.sqlite3-wal
• history.sqlite3

arguably, plugin.msgpackz could also be moved over there, leaving only my personal config files in the Nushell config directory, but that's already out of the scope of this very PR... :wink:

[NotTheDr01ds]

@tombh Apologies for the whiplash here, but ...

We discussed this during our weekly call today, and the conclusion was that we don't think XDG/data-dir is the right way to go here. The main reason is that it becomes difficult for users who want to have their history file alongside config.

I'll try add more info in the morning, but I wanted to redirect you back to an approach more in line with your original thinking in #14434. I haven't reviewed that one in enough detail yet, personally, but I'll try to take a closer look tomorrow.

[gfvioli]

i haven't had a look at the implementation for this but, my personal opinion is that having the following files in ~/.local/share/nushell/ would be cleaner than in ~/.config/nushell/ as right now (my $env.XDG_DATA_HOME | path join nushell and $env.XDG_CONFIG_HOME | path join nushell respectively)

• history.txt
• history.sqlite3-shm
• history.sqlite3-wal
• history.sqlite3

arguably, plugin.msgpackz could also be moved over there, leaving only my personal config files in the Nushell config directory, but that's already out of the scope of this very PR... 😉

I support this comment.

Particularly I think having the history file inside the $XDG_CONFIG_HOME folder can lead to .dotfiles repos saving all commands entered to nushell into the history file by default, which can be problematic if ones not careful.

I would advice to put a warning for this at the very least to add the history file to the .gitignore

[lunagl]

@NotTheDr01ds

We discussed this during our weekly call today, and the conclusion was that we don't think XDG/data-dir is the right way to go here. The main reason is that it becomes difficult for users who want to have their history file alongside config.

I'll try add more info in the morning, but I wanted to redirect you back to an approach more in line with your original thinking in https://github.com/nushell/nushell/pull/14434. I haven't reviewed that one in enough detail yet, personally, but I'll try to take a closer look tomorrow.

I believe the correct solution is clearly to have a sensible default (XDG_STATE_HOME on xdg compliant systems, some data dir otherwise) but also allow customization of the history path, just as it is with config/env file at the moment. I'm not sure why comments in these threads sometimes put those two as being opposing options for solving the issue, and I also don't see why we shouldn't use XDG_STATE_HOME on supported platforms, and whatever else is available on others.

sorry for necroposting, I hope I'm not just causing unnecessary spam, but I felt like it might be useful to consider