`sdata/subcmd-install/3.files-exp.sh` enhancement

[clsty]

What would you like to be added?

Enhance sdata/step/3.install-files.sh in handling

• backup
• copy&overwrite
• output log

Quoting what I said in #2120:

It's ugly because there're several complex snippets (if else thingy) which structure is similar, which means it should better be done via functions.

And most importantly, from the first place, it's not good to generate a pile of *.new or *.old garbages to deal with file conflicts.

I'll find out a better way, for example a customisable syncpattern.conf file containing information about how to deal with each path, each line of which contains two elements, one is file/directory path, another is syncing mode.

How will it help?

A lot.

• #2133
• #2120
• #1863
• #1438
• #1522
  • Talking about fcitx5 though, I think this repo should not touch fcitx5 thingy from the first place, nor should we add fcitx5 as dependency. See also #2141
• https://github.com/end-4/dots-hyprland/issues/1544#issuecomment-3020368549

Extra info

• Currently the script will ask each step before using rsync but the level of automation is not high enough.
• We are not using any symlinks tool, like dotbot. It's super annoying for people only customed to the old-fashioned modification method on dotfiles, according to my previous experience.

[clsty]

Current ideas

yaml config

About how to implement this:

I'll find out a better way, for example a customisable syncpattern.conf file containing information about how to deal with each path, each line of which contains two elements, one is file/directory path, another is syncing mode.

Using a syncpattern.conf containing 2 elements each line is not extendable at all. I think a syncpattern file in *.yaml or *.toml can be used.

If we stick to Bash then yq can be used to parse yaml, but if things get more complicated we maybe need python... Better avoid that though.

version number

Should have a version number inside the yaml configuration file. Everytime it has breaking change the major version number should increase, so that the install script can abort when the major version number is not the same.

Flavor?

As we all know that not all users of this dotfiles use fish or kitty, or windows-like keybindings.

• As for myself, I'm deeply customed to Zsh and Vim-like keybindings. It's just like the water for fish and I can't live without it.
• Other examples are #2175 .
• In fact I has been even asked about why end-4 is not using Vim-like keybindings when I discuss with others in an Arch Linux chat group in Chinese.

Currently we have ~/.config/hypr/custom which is capable if you really need a Vim-like keybindings. However if you also want the latest funcionality provided by the default variant then you'll need to track and sync with the default variant, which means you'll need to manually check the commit history of each file under ~/.config/hypr/hyprland/, which is really time-consuming.

Although I've been thinking a lot that maybe we should provide variants for users to choose, there is an important note that the extra variants should be designed needless or effortless to sync with the default one.

For example if we are adding a Vim-like flavor for keybinds.conf, if that means we will have two versions of keybinds.conf to maintain and sync update from one to another, then it's too annoying to maintain in long term.

Possible useful ideas:

• Using variables for arrow keys (so they can be substituted easily with HJKL)
  • still not perfect considering that HJKL may also be used in default keybinds for other purpose.
• Use script instead of precise command to be bind with keybinds. For example:

  bindd = Super, V, Clipboard history >> clipboard, global, quickshell:overviewClipboardToggle # Clipboard history >> clipboard
  

  When the command has been updated, e.g.

  bindd = Super, V, Clipboard history >> clipboard, global, quickshell:clipboardPanelToggle # Clipboard history >> clipboard
  

  Then the Vim-like variant should also keep up this change, despite that the keybind Super, V itself has not actual update. However, if we write a script e.g. ~/.config/hypr/scripts/clipboard-history-toggle.sh, which content is

  #!/bin/bash 
  quickshell:overviewClipboardToggle
  

  Then whatever the command changes, we always have the keybind configure unchanged:

  $scriptdir=~/.config/hypr/scripts
  bindd = Super, V, Clipboard history >> $scriptdir/clipboard-history.sh # Clipboard history >> clipboard
  

  So that the Vim-like variant will not need an update when the command itself changes.

Track latest changes made easily

A typical need is to track the latest changes of one part of config for example .config/hypr and we need a pratical way to determine the start point of the commit history. For example a user has already catched up with the changes before commit with hash xxxxxx, then we should show the user with the commit history from xxxxxx to HEAD.

[mattvanharn]

Is this something you are currently working on? If, not I am willing to help with this.

[clsty]

Is this something you are currently working on? If, not I am willing to help with this.

Not working on but just ideas.

If, not I am willing to help with this.

Thanks a lot. Since this is much an experimental thing, in 546435db6d7fdee3a69ab72b9baa97393a3fc3df and 3f577f108808fab695723f49a7fe652bd77cc18b I added an option --exp-files for install.sh to call sdata/subcmd-install/3.files-exp.sh. Please leave sdata/step/3.install-files.sh untouched and modify sdata/subcmd-install/3.files-exp.sh and other files needed (e.g. sdata/step/sdata/subcmd-install/3.files-exp.yaml if you are creating a PR.

[clsty]

I recommend firstly implement a simple version. Example: the default yaml config sdata/step/3.install-files.yaml for reference:

version: 0.l
patterns:
  - from: ".config/quickshell"
    to: "$HOME/.config/quickshell"
    mode: sync
  - from: ".config/kitty"
    to: "$HOME/.config/kitty"
    mode: sync
#  ...

The possible values of mode: (by default sync)

• sync: Make the destination completely the same as the source.
• skip: Skip this step
• hard: Overwrite existing files when copying
• soft: Skip existing files when copying
• hard-backup: If target file exists, create backup by renaming it to *.old.<n> where <n> is a number increment from 1 to prevent backup gets overwritten when running twice. (Keep in mind that this script must be idempotent.) Also must compare the actual file content by using MD5 HASH value to prevent generating lots of *.old.<n>s when runnng multiple times. Do not create backup but skip coyping when source and target HASH values are exactly the same.
• soft-backup: If target file exists, copy source file to *.new. Do not create *.new but skip coyping when source and target HASH values are exactly the same.

User can use ./install-files.yaml as custom config. When ./install-files.yaml exists and have correct major version number, merge it together with sdata/step/3.install-files.yaml to generate a cache/install-files.final.yaml to determine how to copy files.

About how to merge two yaml files, I know some software such as rime input method and docker supports a override yaml config, which we may reference from.

[clsty]

As for yaml parser, two options:

• Go: https://github.com/mikefarah/yq
• Python: https://github.com/kislyuk/yq

Better use the Go one because:

• It's much more popular.
• Go has better performance than Python.
• Easy to install: pacman -S go-yq

[mattvanharn]

So if I am understanding this correctly, you intend for the customization options to be read from a config file for each user? Should there be an initial prompt during the install script to create this file if it doesn't exist, or should there be a default config provided and users can edit this config file if they wish to? Having a default config file would be simpler, but it would also require more work from the user to change (if they know about the file at all).

[clsty]

So if I am understanding this correctly, you intend for the customization options to be read from a config file for each user?

Yes.

Should there be an initial prompt during the install script to create this file if it doesn't exist,

No. If it does not exist just ignore it. If it exists and have a mismatch major version number then prompt error and abort immediately.

or should there be a default config provided and users can edit this config file if they wish to?

Use the default config shipped inside this repo directly. User should not edit it but create their own custom yaml config as override.

About yaml override for example see https://github.com/mikefarah/yq/discussions/1437

[mattvanharn]

I was able to get the YAML file and the YAML parser working using yq. I'm now adding the functionality of the original 3.install-files.sh to the experimental file and filling out the default YAML file, and then I'll try and figure out how to do the user customization options.

As for creating the override file, I do think a prompt could be useful. There is going to need to be a prompt anyway to notify users that they need to create this file if they want options other than the default, but some people might not know how to use YAML or might do it incorrectly leading to errors and frustration. Using a script which creates the override file for the users could make that a lot simpler for the user, or there at least could be an initial prompt if the file doesn't exist.

[clsty]

Update: I've reworked the backup_config function so it's at least a bit better than before.

@mattvanharn I agree that the script can have a prompt creating user yaml config, though it should not be tracked by git (use .gitignore probably). If you'd like to submit a PR, go ahead please.

[mattvanharn]

Sorry, been a little busy. I will submit the PR tomorrow, my script still has some bugs I need to fix.

[clsty]

Update: with the latest version if you're to test sdata/subcmd-install/3.files-exp.sh, just run

./setup install-files --exp-files

[mattvanharn]

I just submitted the PR with my experimental script as well as the YAML file. As far as I know it covers the following issues discussed in this thread:

• YAML config file
• Interactive user preferences
• Multiple sync modes: sync, soft, hard, hard-backup, soft-backup, skip, skip-if-exists
• MD5 hash comparison for idempotent backups
• Complete coverage of all config directories and files from original script

The script is not complete yet though. Some future work that needs to be done is handling the keybind preferences, but that seemed like a separate issue so I did not include that. Also, I am still new to yq so I am sure there are plenty of areas which could be simplified or improved by using yq syntax instead of bash (yq also needs to be included as one of the packages downloaded during the install process). Finally, I am not sure what the default configuration should be so feel free to make any adjustments to that in the YAML file.

If you have any questions on this or would like more help on the rest of this, I would be happy to help.

[eslam-allam]

How do you guys track modifications you make to config files after installing? Caelestia for example uses symlinks. This lets me fork their repo, modify what ever I want over time after installation, then just commit to my fork. This way I can reinstall on a new pc with and keep all my changes. The only was I can see this working in this repo is to modify the files in the source folder then run the installer / copy that specific file to the .config folder. I'm really curious if anyone else is experiencing this problem.

[clsty]

How do you guys track modifications you make to config files after installing? Caelestia for example uses symlinks. This lets me fork their repo, modify what ever I want over time after installation, then just commit to my fork. This way I can reinstall on a new pc with and keep all my changes. The only was I can see this working in this repo is to modify the files in the source folder then run the installer / copy that specific file to the .config folder. I'm really curious if anyone else is experiencing this problem.

I'm not a fan of symlink style but I guess we can make it one of the sync mode in the yaml config.

[clsty]

I've updated the TODOs in 3.files-exp.sh.

[clsty]

I will be kinda busy in near future so the progress on this will be slowed down, however PRs are welcomed. Thanks for your help.