Apply fails if using externals with non-existent/ignored subfolder in chezmoiignore

[marcus-crane]

Describe the bug

I don't quite understand what is going on exactly with this bug since I haven't read enough about the order of operations with Chezmoi but I can at least explain what appears to be happening:

This bug occurs when the following is true:

• A folder is created as part of the .chezmoiexternal process (perhaps possible through normal means?)
• A subfolder that does not normally exist (but will exist post-checkout) is referenced in .chezmoiignore

When running chezmoi apply, Chezmoi will start to check out the external folder but seemingly when it gets to a file that is ignored (.oh-my-zsh/cache) in this case, it does an LSTAT for the folder within my home directory, finds that it doesn't exist and bails out with an error.

Within the repo, .oh-my-zsh/cache contains a .gitkeep file so it seems that Chezmoi's state registering portion is seeing the .gitkeep but not realising it isn't committed to disk and then when it goes to check, it fails because the cache folder also doesn't exist and so lstat can't successfully check the file without the full path existing?

Anyway, it's a minor issue that I never run into, and can be fixed by just manually creating the ignored folder, but as I'm setting up a new environment, I'm running into it for the first time.

Thanks!

To reproduce

$ pwd
/Users/marcus/.local/share/chezmoi
$ cat .chezmoiignore
.oh-my-zsh/cache
$ cat .chezmoiexternal.toml 
[".oh-my-zsh"]
    type = "archive"
    url = "https://github.com/ohmyzsh/ohmyzsh/archive/master.tar.gz"
    exact = true
    stripComponents = 1
    refreshPeriod = "168h"
$ chezmoi apply

Expected behavior

A clear and concise description of what you expected to happen.

Output of command with the --verbose flag

$ chezmoi --verbose --debug apply
[... many lines later ...]
2022-11-26T15:49:04+13:00 INF Lstat component=system name=/Users/marcus/.oh-my-zsh/LICENSE.txt
2022-11-26T15:49:04+13:00 INF Get bucket=entryState component=persistentState key=/Users/marcus/.oh-my-zsh/LICENSE.txt value=
2022-11-26T15:49:04+13:00 INF ReadFile component=system data="MIT License\n\nCopyright (c) 2009-2022 Robby Russell and contribut..." name=/Users/marcus/.oh-my-zsh/LICENSE.txt size=1142
2022-11-26T15:49:04+13:00 INF Set bucket=entryState component=persistentState key=/Users/marcus/.oh-my-zsh/LICENSE.txt value="{\n  \"type\": \"file\",\n  \"mode\": 420,\n  \"contentsSHA256\": \"5b6e033667dba169de6213440dea2d7f2286e679c1f56a3fcb01103d2bcb306d\"\n}\n"
2022-11-26T15:49:04+13:00 INF defaultPreApplyFunc actualEntryState={"ContentsSHA256":"5b6e033667dba169de6213440dea2d7f2286e679c1f56a3fcb01103d2bcb306d","Mode":420,"Type":"file","contents":"MIT License\n\nCopyright (c) 2009-2022 Robby Russell and contribut..."} lastWrittenEntryState={"ContentsSHA256":"5b6e033667dba169de6213440dea2d7f2286e679c1f56a3fcb01103d2bcb306d","Mode":420,"Type":"file","contents":"MIT License\n\nCopyright (c) 2009-2022 Robby Russell and contribut..."} targetEntryState={"ContentsSHA256":"5b6e033667dba169de6213440dea2d7f2286e679c1f56a3fcb01103d2bcb306d","Mode":420,"Type":"file","contents":"MIT License\n\nCopyright (c) 2009-2022 Robby Russell and contribut..."} targetRelPath=.oh-my-zsh/LICENSE.txt
2022-11-26T15:49:04+13:00 ERR Lstat error="lstat /Users/marcus/.oh-my-zsh/cache/.gitkeep: no such file or directory" component=system name=/Users/marcus/.oh-my-zsh/cache/.gitkeep
2022-11-26T15:49:04+13:00 INF Get bucket=entryState component=persistentState key=/Users/marcus/.oh-my-zsh/cache/.gitkeep value=
2022-11-26T15:49:04+13:00 INF defaultPreApplyFunc actualEntryState={"ContentsSHA256":"","Mode":0,"Type":"remove"} lastWrittenEntryState={} targetEntryState={"ContentsSHA256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","Mode":420,"Type":"file"} targetRelPath=.oh-my-zsh/cache/.gitkeep
2022-11-26T15:49:04+13:00 ERR Lstat error="lstat /Users/marcus/.oh-my-zsh/cache/.gitkeep: no such file or directory" component=system name=/Users/marcus/.oh-my-zsh/cache/.gitkeep
diff --git a/.oh-my-zsh/cache/.gitkeep b/.oh-my-zsh/cache/.gitkeep
new file mode 100644
index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
--- /dev/null
+++ b/.oh-my-zsh/cache/.gitkeep
2022-11-26T15:49:04+13:00 ERR WriteFile error="stat /Users/marcus/.oh-my-zsh/cache: no such file or directory" component=system data= name=/Users/marcus/.oh-my-zsh/cache/.gitkeep perm=420 size=0
chezmoi: stat /Users/marcus/.oh-my-zsh/cache: no such file or directory

Output of chezmoi doctor

$ chezmoi doctor
RESULT    CHECK                MESSAGE
ok        version              v2.27.2, commit 882d0808feb1fc8112b411ed2216f31306656861, built at 2022-11-24T21:50:16Z, built by Homebrew
ok        latest-version       v2.27.2
ok        os-arch              darwin/amd64
ok        uname                Darwin Marcuss-Mini 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct  9 20:14:54 PDT 2022; root:xnu-8792.41.9~2/RELEASE_X86_64 x86_64
ok        go-version           go1.19.3 (gc)
ok        executable           /usr/local/bin/chezmoi
ok        upgrade-method       replace-executable
ok        config-file          ~/.config/chezmoi/chezmoi.toml, last modified 2022-11-22T20:44:10+13:00
warning   source-dir           ~/.local/share/chezmoi is a git working tree (dirty)
ok        suspicious-entries   no suspicious entries
warning   working-tree         ~/.local/share/chezmoi is a git working tree (dirty)
ok        dest-dir             ~ is a directory
ok        umask                022
ok        cd-command           found /bin/zsh
ok        cd-args              /bin/zsh
info      diff-command         not set
ok        edit-command         found /usr/bin/vi
ok        edit-args            /usr/bin/vi
ok        git-command          found /usr/bin/git, version 2.37.1
ok        merge-command        found /usr/bin/vimdiff
ok        shell-command        found /bin/zsh
ok        shell-args           /bin/zsh
info      age-command          age not found in $PATH
info      gpg-command          gpg not found in $PATH
info      pinentry-command     not set
ok        1password-command    found /usr/local/bin/op, version 2.7.2
info      bitwarden-command    bw not found in $PATH
info      gopass-command       gopass not found in $PATH
info      keepassxc-command    keepassxc-cli not found in $PATH
info      keepassxc-db         not set
info      keeper-command       keeper not found in $PATH
info      lastpass-command     lpass not found in $PATH
info      pass-command         pass not found in $PATH
info      passhole-command     ph not found in $PATH
info      vault-command        vault not found in $PATH
info      secret-command       not set

Additional context

Add any other context about the problem here.

[twpayne]

Thank you for the comprehensive bug report!

I think what's happening here is that there's a conflict between .chezmoiignore and .chezmoiexternal which chezmoi is not handling correctly.

I think you can work around this in the short term by putting

.oh-my-zsh/cache/**

in your .chezmoiignore file instead of .oh-my-zsh/cache.

[miquecg]

I think you can work around this in the short term by putting

.oh-my-zsh/cache/**

Not sure it solves the problem. This rule will ignore all contents but the folder itself, thus also the .gitkeep file that actually exists in the tar file. Is that a problem with exact = true? To my mind final state should be cache/.gitkeep but everything else ignored.

Today I've had the same issue when doing chezmoi apply ~/.oh-my-zsh and noticed that having the rule caused chezmoi status | grep -i cache to not list the folder. I had to remove it, apply changes and then add it back again.

[twpayne]

For what it's worth, I would have expected the .chezmoiignore contents of:

!.oh-my-zsh/cache/.gitkeep
.oh-my-zsh/cache/**

to mean that .oh-my-zsh/cache/.gitkeep is created but all other files in .oh-my-zsh/cache are ignored, but this is not the case. So, I think this is a bug.

[AnhQuanTrl]

I have encountered this exact issue today by trying to apply my dotfile on a new system. Any ideas how to fix this? Creating empty directories in destination works but it is kinda hacky.

[twpayne]

Any ideas how to fix this?

Yes. Rework the internals of chezmoi's state representation of the target state to use a proper tree structure and populate implicitly-created nodes with virtual markers. After the source state is read, replace any remaining virtual markers with directory nodes with default permissions. Please follow the contributing guidelines when you do this.

[twpayne]

Well it turns out that I was completely wrong on the effort required to fix this. After #3705 which involved externals, I took another look at this and it turns out it was a missing special case for ignoring directories in externals. Should be fixed with #3717.