Feature request: XDG Basedir Compliance

[jasonkarns]

• YubiKey Manager (ykman) version:

$ ykman -v
YubiKey Manager (ykman) version: 3.1.1
Libraries:
    libykpers 1.20.0
    libusb 1.0.23

• How was it installed?: homebrew
• Operating system and version: macOS Catalina 10.15.3
• YubiKey model and version:

ykman info
Device type: YubiKey 5C Nano
Firmware version: 5.1.2
Form factor: Nano (USB-C)
Enabled USB interfaces: OTP+FIDO+CCID

• Bug description summary:

ykman writes to ~/.ykman/ (at least ~/.ykman/oath.json) but should be writing to XDG-compliant directories. (Especially when XDG_* env vars are set.)

I can't speak to all the different kinds of files and types of data that ykman writes, but the appropriate data would ideally be written to Data, Config, Runtime and Cache directories per the XDG specification.

[dagheyman]

Thanks for the suggestion, we'll consider it for the future!

[gbdlin]

I'm working on PR for that and I need some confirmation of current logic and if we want preserve some part of it.

As far as my understanding goes, currently ykman will search for the .ykman directory in the location from which the command was invoked. If it doesn't find one, it will look for one in user home directory. With that said, I have 2 questions:

1. What is the use case for local .ykman directory? Is this documented anywhere right now? Do we have any confirmation that somebody is using it and we should keep that logic intact?
2. Do we want to move config files from ~/.ykman to new location, compliant with XDG Basedir?
3. How should we approach this on platforms not covered by XDG, for example Windows?

[gbdlin]

For now I've implemented it with assumption that:

1. local .ykman directory stays as is.
2. ~/.ykman directory is not being moved to new location
3. when ~/.ykman directory exists, will be used instead of XDG compliant location
4. when ~/.ykman directory doesn't exist, XDG compliant location will be used
5. No special behavior for other platforms (like currently)

Also, I've introduced pathlib.Path instead of os.path functions usage.

See #393

[dainnilsson]

Thanks! I've merged the PR with some additional changes:

1. I removed the use of ./.ykman since I don't think it's used and doesn't really make sense (and now XDG_CONFIG_HOME can be used to customize the location).
2. I didn't want $XDG_CONFIG_HOME/ykman to be the default for creation, especially for Windows/Mac, so there is now a check to see if the base directory of $XDG_CONFIG_HOME already exists before choosing that location.

[jasonkarns]

I'm confused by this:

I didn't want $XDG_CONFIG_HOME/ykman to be the default for creation, especially for Windows/Mac, so there is now a check to see if the base directory of $XDG_CONFIG_HOME already exists before choosing that location.

Does this check respect the spec such that $HOME/.config is the default directory if XDG_CONFIG_HOME is empty or unset? I hope you mean that you check for the directory: ${XDG_CONFIG_HOME:-$HOME/.config) and not the presence of the variable correct?

I'm also a little confused by the desire to not do this for mac users. I'm a mac user and specifically requested this feature. And I would wager that virtually 100% of mac-using yubikey users have a $HOME/.config directory (if not a customized XDG_CONFIG_HOME). This assumption is because most yubikey users are likely to be technical, and would therefore likely have other cli tooling installed on their macs; at least one of which would very likely conform to the XDG spec. (every git user, for instance)

My confusion, however, is why windows or mac should be excluded from this behavior. Both windows and mac users have $HOME. And the XDG spec defaults to using $HOME for the configuration. Now I could understand excluding windows if the windows conventions of %APPDATA% are used. But if windows is excluded and still writing to %HOME%, then I'm at a loss.

edit disregard this. I see that XDG should be respected as long as the directory exists: https://github.com/Yubico/yubikey-manager/commit/62342ebb711c0a2134e397ef1d46fb07002019e9#diff-f17b3a89689496e9b2ad5c1a2a11f967a30e423fa3f351e871fc97cff872bdccR54

[jasonkarns]

Thanks @gbdlin and @dainnilsson for working on this!!

[gbdlin]

I removed the use of ./.ykman since I don't think it's used and doesn't really make sense (and now XDG_CONFIG_HOME can be used to customize the location)

I don't think this is serving the same purpose, as XDG_CONFIG_HOME, but I understand removing the ./.ykman support.

I didn't want $XDG_CONFIG_HOME/ykman to be the default for creation, especially for Windows/Mac, so there is now a check to see if the base directory of $XDG_CONFIG_HOME already exists before choosing that location.

I agree with @jasonkarns on that. Mac OS is also affected by XDG Basedir and there is no harm using ~/.config on windows machines (as probably a lot of other software uses it if they don't care about using Windows-specific directories). Also it yields some inconsistent results between systems if presence of this directory decides on the file path chosen by ykman to store the config file.

[dainnilsson]

Scratch Mac from what I said then, it's more about not forcing non-XDG systems into that behavior. The rational here is that a "XDG user" will have a ~/.config (or whatever XDG_CONFIG_HOME points to) already, and then it will be used. I did some very un-scientific research by looking at a few different Windows PC's and none of them had a ~/.config/, though there were several ~/.appname directories.

It's not ideal that the storage location when no configuration exists will depend on the existence of a directory which may be created by some other app, but I don't really see it causing problems. I think this is an OK compromise?

[gbdlin]

As this project can be used both as a library and a CLI tool, there are very high changes it being used somewhere in the enterprise/automated environment. Some automation may then look for the file in one location but not in the other, which would be fine if ykman's behavior would be consistent and affected only by the existence of files and directories specific to it.

With your solution, this may become a very hard to debug case of why this tool is choosing to save configuration in different place when the actual config file doesn't exist in either of those locations, so it shouldn't be affected by some fallback for backwards compatibility.

On the other hand, I can't find any harm in creating .config directory and using it on any Windows environment, as ykman is not using Microsoft's standard location for configuration files anyway. It may be actually appreciated by Windows users, because it results in reduced clutter in their home directory, as files starting with . are not hidden by default on Windows.

[jasonkarns]

I haven't been following the code closely, but my question is: what happens if a user installs and uses ykman on a brand new system (ie, XDG_HOME_CONFIG directory doesn't exist). ykman would write to ~/.ykman; then later they install git or any other tool that creates ~/.config for the first time.

Where will ykman read from the next time it runs?

[gbdlin]

@jasonkarns ykman first looks for the config files in all known locations in order:

• ~~./.ykman~~ - This location was removed with current changes
• ~/.ykman
• ${XDG_CONFIG_DIR}/ykman (if variable is not set, it defaults to ~/.config) - this location was added with current changes.

If no location is found, it will check for ${XDG_CONFIG_DIR} existence (again, taking default variable value) and if this directory exists, ykman will create sub-directory in it and use that, if it doesn't exist, it will store configuration in ~/.ykman directory.

On 2nd run, it will discover existence of ~/.ykman directory first and read the configuration from it, as it is higher on the list. Of course it will happen only if any configuration was saved in the first run, which will, for now, happen only when user tells ykman to remember his password for OATH application.

[dainnilsson]

I see the different options as follows:

1. Do nothing, no XDG support.
2. Allow XDG through explicit opt-in (opt-in has to be specific to ykman).
3. Allow XDG through implicit opt-in (opt-in can be based on system state).
4. Default to XDG, but use ~/.ykman/ if it exists.
5. Use only XDG (ignore ~/.ykman if it exists).

1 is the current ykman 3 behavior, which is why this issue exists in the first place. 3 is the behavior currently committed. 4 is the behavior of the initial PR.

The issues around inconsistencies due to non-ykman configuration that @gbdlin bring up are valid, but are applicable to 4 as well. In that scenario we have to accommodate the fact that some users will have configuration (if they've saved at least one OATH password) and other won't. After updating to behavior 4, anyone who saves configuration for the first time will end up with ~/.config/ykman, where others have ~/.ykman. In both cases 3 and 4 it can be confusing/surprising for the user.

Based on this, I'm leaning more and more towards option 2. This is fully backwards compatible, with no unexpected behavior. It allows for XDG for anyone who wants it. The down-side is that it won't do XDG by default.

I propose the following logic to implement option 2:

If $XDG_CONFIG_HOME/ykman exists, use it. If not, use ~/.ykman.

This would mean that anyone wanting to use the XDG layout can explicity opt in by creating an empty directory: mkdir ~/.config/ykman, or by moving existing config: mv ~/.ykman ~/.config/ykman.

We aim to release a new version very soon (next week) so I'm very much trying to err on the side of caution here and not make any drastic changes. We may re-evaluate this at a later point, but for now I think the "explicit opt-in" approach gets us pretty far.

[gbdlin]

The difference between option 3. and option 4. is: option 4 is an explicit opt-out (opt out is specific to ykman), but with default opt-out for users who have saved at least one OATH in previous versions of ykman and that's IMO the best option, as it defaults to XDG basedir compliance for any new user of this tool and leaves things unchanged for old users just for backwards compatibility. 2. is for sure a step backwards.

Be aware that most users who want XDG compliance in most cases either wants their home directory to be less bloated or wants to have an easy way to back up things in a sorted and controlled manner. Both reasons can come from some situation that already happened (home directory is already a mess or, even worse, configuration for some tools is not backed up properly), so in my understanding, solution for those problems should be opt-out instead of opt-in.

[dainnilsson]

The more I look into this the more unanswered questions pop up. We have a new major release coming up imminently, and I don't want to rush into anything here that we then have to back out of. I'm not sure if XDG_CONFIG_HOME is the correct location for the current contents of ~/.ykman. I think XDG_DATA_HOME might be more correct, which may mean we need to have two separate locations instead of just one (assuming we add anything that does belong in CONFIG). We should also consider if XDG is what we should do for all platforms, or if we should use %APPDATA% on Windows, follow https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.html#//apple_ref/doc/uid/TP40010672-CH2-SW1 on Apple, and so on.

The general strategy I'm suggesting here is as follows:

We start by adding experimental (unsupported, subject to change at any time) support for XDG layout in a way that is fully opt-in. We can use this to hammer out any quirks and get feedback without committing to any specific behavior. I'm thinking we use an environment variable for this, something like YKMAN_XDG_EXPERIMENTAL=1 to enable. Without this set, the current behavior of using ~/.ykman remains as before.

We figure out if we should do anything Platform specific, what we need to do to ensure existing users aren't too negatively affected, and so on.

Once we're satisfied with the behavior we can move to make it the default, possibly with some fallback or automatic migration. Longterm I would want to drop any such fallback.

To implement this swiftly, I've committed an alternative minimum implementation for inclusion into ykman 4.0.0 (https://github.com/Yubico/yubikey-manager/commit/980e0af9367ffda494cc04549f649394eb3539a5). The main branch will continue development based on the current state, and we'll have more time to make sure we get things right.

[dainnilsson]

ykman 5.0 is now released, and follows the XDG basedir guidelines.