jj-vcs
Previously ignored files should not become tracked on checkout
Description
When updating from a commit where some files are ignored to a commit where they are not ignored (typically an older version), the ignored files will be picked up by subsequent snapshotting. That's rarely what one wants, and it's especially problematic if the ignored files are many (e.g. target/ in Rust projects).
Steps to Reproduce the Problem
cd $(mktemp -d)
jj git init
jj new
echo ignored > .gitignore
touch ignored
jj edit @-
jj status
Actual Behavior
The ignored file is tracked.
Expected Behavior
The ignored file does not get tracked. Perhaps jj status reports that it's in some kind of intermediate state.
We could implement it by having the WorkingCopy keep track of this type of file. Files would enter this state when updating the working copy and the ignored paths change. They would exit it when they're either either deleted or properly ignored again.
This idea was originally suggested by @ony.
Specifications
- Platform: All
- Version: 0.25.0 (and probably all before it)
Duplicates of https://github.com/jj-vcs/jj/issues/3204?
Files would enter this state when updating the working copy and the ignored paths change. They would exit it when they're either either deleted or properly ignored again.
fwiw, git submodule directories are processed in a similar manner. They're marked as submodule (= ignored) internally.
Duplicates of #3204?
Ah, that's the issue I was looking for! I spent at least 5 minutes searching for it (e.g. "tracked ignored" doesn't work). I guess I'll mark that one as duplicate instead so it will be easier to find the issue (i.e. this one) next time.
In case it's helpful, I ran into this when I tried to use the git pattern of adding a (.env) file in one commit and then adding it to .gitignore in a subsequent commit. (By the way -- thanks for jj! It's great.)
I just pushed a secret to a remote due to this, it was only a test one though so no biggies 😄
Basically, I was working on something that required me to temporarily put a secret in a file for test purposes and I took special care to .gitignore it, knowing that jj would otherwise start to track it automatically. A few days later (today), someone asked me about some code I had written in some sibling commit so I jj edited it and pushed it up to a temporary branch. I did not look at what I would push, because I just wanted to quickly share code with someone else (actually thought "because it is jj I do not even have to add any changes, I can just quickly do them and push them..."). Because that sibling commit did not include the changes to .gitignore and the untracked secret was moved over when I ran jj edit, I ended up pushing it.
I guess what I am saying is that whatever comes out of this issue, should have either warned me when I ran jj edit that some files will no longer be ignored or should have ignored them automatically again. Notably, it should not just delete that file, because I would like to still have it around when I work on the original change again. I think this is what the issue description is suggesting anyway, I am just repeating it here explicitly 👍
I was thinking about this issue and why it is so vexing for me, and I thought of this principle:
jj edit x, followed byjj edit yto go back to where I was before, should have no effect on the contents of changes x and y
This problem underlying this issue is about that principle being violated. The ignored-ness of the file is one way to look at it (since I think that is the only way for this to break?) but another way of looking at it is that immediately after running jj edit y the resulting on-disk state (as in what would be captured in the next jj snapshot) does not match the contents of y that I just requested.
I started using JJ today and this issue was the first one that I ran into with existing git repositories. In some I had legacy files that I kept around but hadn't put on the ignore list. JJ picked them up immediately and then I struggled with defining ignore lists on the main branch. The fix on the main branch led to this issue when I switched to another branch that didn't have the ignore list yet.
One work around that comes to my mind is git's internal ignore list that is kept inside the .git folder and is not tracked by version control. If there was something similar in JJ, I could work around the issue by manually copying the current ignore list into this special list. Definitely not perfect, however it feels like the issue is caused by JJs architecture and therefore difficult to solve.
jj respects .git/info/exclude already. (Note that adding stuff there won't untrack it if jj has already picked it up; you'll need to jj file untrack it or abandon @.)
For accidentally added files, is there a way to comb through the oplog and filter them out? I guess it's not possible since they get added as a side effect of other commands right? I ran into this when a vscode lsp extension generated ~1000 build files, which now makes commands such as jj operation log --patch difficult to use. It would be nice if there was a way to solve this without needing to delete + clone the repo again!
~1000 build files, which now makes commands such as
jj operation log --patchdifficult to use. It would be nice if there was a way to solve this without needing to delete + clone the repo again!
If you're okay to nuke these operations, you can use jj op abandon <first-bad-op>-..<last-bad-op>.
Oh, I totally had the wrong mental model of jj op abandon 😅 I thought changing it would actually modify history, like unapplying the change made at that step, but its just erasing the history entries, that makes sense! I still had the problem of operations being slow and .jj growing quite large. I tried doing jj op abandon ..@- and jj util gc --expire now, which did not really help. I guess the objects still exist in the evolog of the change where they were accidentally committed, so they don't get collected (like the documentation mentions)? For what it's worth i really like the auto-tracking behavior in general, its just unfortunate how it interacts with .gitignores currently.
I had those changes in abandoned change. And for me jj op abandon just got them moved to a different operations. I guess, that's what re-parenting descendant operations means.
Now they seems like they squashed into descendant operations. E.g. I have real operation in useful history path mixed together with other hidden changes.
Maybe if I would had in evolog for that change untracking those files then squashing operations that add and then remove those files for that change would collapse them like.
jj git init && echo touch unwanted_file && jj st && rm unwanted_file && jj op log -p
Will end up with fe187811c232 followed by removal
@ 3cfe685f8923 someone@somewhere 29 seconds ago, lasted 2 milliseconds
│ snapshot working copy
│ args: jj st
│
│ Changed commits:
│ ○ + nkvmpzkx 20a25ea9 (empty) (no description set)
│ - nkvmpzkx hidden e2b93bba (no description set)
│ Removed regular file unwanted_file:
│ (empty)
│
│ Changed working copy default@:
│ + nkvmpzkx 20a25ea9 (empty) (no description set)
│ - nkvmpzkx hidden e2b93bba (no description set)
○ fe187811c232 someone@somewhere 38 seconds ago, lasted 3 milliseconds
│ snapshot working copy
│ args: jj st
│
│ Changed commits:
│ ○ + nkvmpzkx e2b93bba (no description set)
│ - nkvmpzkx hidden c8da39b5 (empty) (no description set)
│ Added regular file unwanted_file:
│ (empty)
│
│ Changed working copy default@:
│ + nkvmpzkx e2b93bba (no description set)
│ - nkvmpzkx hidden c8da39b5 (empty) (no description set)
...
And then jj op abandon fe187811c232. Then it really disappears
jj op abandon @- && jj op log -p
@ e1b8558038fd someone@somewhere 1 minute ago, lasted 2 milliseconds
│ snapshot working copy
│ args: jj st
│
│ Changed commits:
│ ○ + nkvmpzkx 20a25ea9 (empty) (no description set)
│ - nkvmpzkx hidden c8da39b5 (empty) (no description set)
│
│ Changed working copy default@:
│ + nkvmpzkx 20a25ea9 (empty) (no description set)
│ - nkvmpzkx hidden c8da39b5 (empty) (no description set)
○ cf7a6710ba57 someone@somewhere 2 minutes ago, lasted 3 milliseconds
│ add workspace 'default'
...
Apparently same works with abandoning (at least on jj v0.29.0):
jj git init && echo test > unwanted_file && jj abandon && jj op log -p && jj op abandon @- && jj op log -p
But if there are more operations in between, have to abandon all of them before one that effectively drops files either via abandoning change or via removing/untracking
jj git init && echo test > unwanted_file && jj new -B @ && jj abandon @+ && jj op log -p && jj op abandon @-- && jj op log -p
One op abandon will not remove mention of unwanted_file in op log
@ 436bcdedacf7 someone@somewhere 20 seconds ago, lasted less than a microsecond
│ abandon commit 51677054d03a918ed94c41bcc6a1574b86805f18
│ args: jj abandon '@+'
│
│ Changed commits:
│ ○ - xrynvlkw hidden 51677054 (no description set)
│ Added regular file unwanted_file:
│ 1: test
○ 1e1a4f5e0a64 someone@somewhere 20 seconds ago, lasted less than a microsecond
│ new empty commit
│ args: jj new -B @
│
│ Changed commits:
│ ○ + xrynvlkw 51677054 (no description set)
│ │ - xrynvlkw hidden 20b67e9e (empty) (no description set)
│ │ Added regular file unwanted_file:
│ │ 1: test
│ ○ + sxtoskqv 5087541f (empty) (no description set)
│
│ Changed working copy default@:
│ + sxtoskqv 5087541f (empty) (no description set)
│ - xrynvlkw hidden 20b67e9e (empty) (no description set)
(notice 1e1a4f5e0a64 includes both addition of file and result of jj new -B @)
I had to do another abandon/squash jj op abandon @- on top to squash it with abandoning operation and collapse addition of that file.
P.S. It is a pity that args: not accumulated with ancestor operations "squashed" down. E.g.
I'd like to see args jj new -B @ && jj abandon '@+' instead of just jj abandon @+ or sequence of them.
P.P.S. Maybe jj op abandon @- indeed uses confusing naming since nothing is abandoned (aside from args:) like in case of jj abandon @- (changes snapshot of @). It works more like jj squash --from @- (preserves snapshot of @).
jj respects
.git/info/excludealready. (Note that adding stuff there won't untrack it if jj has already picked it up; you'll need tojj file untrackit or abandon @.)
Can we have "auto untrack" for gitignored files too? The way I understand this issue is that it applies to editing previous commits. But
jj git init
jj new
touch ignored
jj status
echo ignored > .gitignore
jj status
still produces
A .gitignore
A ignored
Expected: ignored should not be tracked.
Auto-tracking is good. I just want it to also untrack gitignored files dynamically (including with watchman).
Hello! I hit my head quite hard on this today, twice.
I had a folder foo tracked by Git, with a sub-folder bar also tracked by Git, i.e. the following directory structure:
foo/
foo/.git
foo/.gitignore
foo/bar
foo/... more files
foo/bar/.git
foo/bar/... more files
I also had a foo/.gitignore file with bar listed in it. Importantly, this was not staged. A little after setting up Jujutsu in foo with jj git init --colocate, I ran jj new --before @. This made a new commit -- from the perspective of a previous commit that did not have .gitignore. This meant that Jujutsu began tracking all of bar upon the next invocation of a Jujutsu command, as a result. I then ran jj undo and all my files in bar were deleted. At this point I started freaking out a little bit, because I didn't notice my files were deleted until a bit later, and restoring to various commits in the reflog didn't restore them.
(The Jujutsu Discord kindly helped me restore them. jj op restore to the commit that snapshotted them, directly after jj new --before @ when jj log was run, worked. I believe restoring to every other operation state in the oplog would not restore them, however. This was really very scary because though the subdirectory was a git repo, I had a bunch of unstaged changes.)
Just wanted to give this as a testimony towards changing the current behavior! I think the current behavior is consistent and predictable if you're thinking about it. I also think it is weird and unusual, and a bit of a footgun, especially coming from Git. Even though it seems quite hard to actually lose your work this way (would you have to run jj op abandon? not sure) it's quite scary if you don't know what's going on! And even if you do know what's going on, it's annoying -- I accidentally added a bunch of stuff I didn't want to in a different repo earlier, though there it was more obvious what had happened and how to proceed.
My suggestion for a fix (not knowing anything about Jujutsu internals) would be to check on every operation that switches commits whether the set of currently ignored files would still be ignored, and if they're not then fail and force the user to pass a --force flag to actually perform the operation. I think this is purely an issue of the user not thinking hard about what Jujutsu is doing, so I think a straightforward solution is to tell the user to think twice.
The current behavior around ignore files seems consistent with Jujutsu's mental model, but it's unexpected if you're not thinking about it.
I think this is purely an issue of the user not thinking hard about what Jujutsu is doing, so I think a straightforward solution is to tell the user to think twice.
No. You should be able to use jj edit aaaaaaaa, then jj edit bbbbbbbb (where that was your previous active change) and it should be a no-op (or jj new aaaaaaaa, and the resulting change will always be empty).
There are some good proposals in #323, such as https://github.com/jj-vcs/jj/issues/323#issuecomment-2571760838. I have a similar idea also, maybe I'll write a comment describing it.
I would note that the comment you link to is for a (complicated) outline of an overhaul of the existing Jujutsu system, while a check of the currently ignored files + a new flag can be implemented now. (And removed later if desired.)
Granted, this only accomplishes "Correctness: don't lose work when switching commits/merging with new working copy".