Auto-add of untracked files screws me up every time

[durin42]

Description

Initially I thought the auto-add of files was a neat idea, but in practice I just leave untracked files in my repo all the time, and tools like patch(1) assume they can drop .orig or similar files in the WC without it being a problem. I think every time I've used jj I've ended up getting grumpy at auto-adds and having to rip something out of a commit, sometimes after doing a push (when it was effectively emulating hg import for example).

Steps to Reproduce the Problem

1. patch -p1 < some.diff (or similar)
2. jj describe && jj close && jj git push
3. Look at web view of diff, notice you committed a .orig file again (or similar)

Expected Behavior

I (still) don't expect auto-adds, and it really surprises me every time, plus it's super frustrating to have to ignore every file spec I might create as a temporary scratch file or what have you.

Actual Behavior

As above.

Specifications

• Platform: macOS
• Version: 6c83eb6ae3c56f1912a630dca4751c7f723a81a3

[arxanas]

Agreed on that — I think automatically committing tracked files is fine, but untracked is probably bad:

• They could be very big
• They could contain secrets
• Querying the working copy for only tracked files is probably more efficient in practice (like with git status -uno)

[martinvonz]

I mostly find the feature useful, but I also agree that it can be annoying and confusing. The worst case I've noticed is when you - perhaps accidentally - check out the root commit, where there's no .gitignore file containing target/. Then any command you run will try to commit GBs of data. I ran into that again just a few days ago, and was actually thinking of adding a config for the auto-add behavior.

When looking at an old version of the repo, you'll not see untracked files (e.g. jj --at-op=<some old operation> status), but that seems fine. I think the most annoying bit will be to add a way of providing that information to users who want to commit the working copy in the background (like we will probably do at Google). I'll probably skip that bit to start with when I implement this.

[tp-woven]

Just to throw my 2c in: With the exception of new files that are also added to .gitignore, I actually really like the automatic tracking. I also think the same "repro" from the first comment can be used as a reason to auto-track - if you don't jj st before pushing, you're just as likely to miss a file that you should have added as you are a file that you should have ignored. So my preference is that this would be configurable if possible, rather than just removed.

[elasticdog]

Just wanted to say that I really like the auto-tracking feature as well. After using jj for a while, going back to manually having to think about adding files to be tracked seems like a lot of extra work and possibly more error prone. I'm already used to checking jj status to make sure that I'm committing the expected files...that said, I can definitely see how not thinking about the appropriate ignores right away could be troublesome if the files are large, and also acknowledge the the root commit situation (although I don't know how often that would realistically come up in day-to-day usage).

[necauqua]

The thing with accidentally committing GBs of target when you forgot to ignore it or checkout some old commit is that there's no GC at the moment, so those GBs are forever in the history.

Especially if you have something automatically creating it, like direnv, the moment you check out the commit before your update of gitignore, literally happened to me with jj repo more than once.

The only way is to recreate the repo (losing the oplog) this time not forgetting to ignore stuff/not editing old commits - which is kind of meh as well.

Full GC of course means basically the same thing, only automated by a single command, but something like jj undo --harder [--at-op OP] [--and-immediate-gc-too] (are you sure [y/N]) to delete the last op/OP unrecoverably and gc things it referenced could be cool.

[ony]

Maybe it is possible to track ignores in jj too. If you switch away from commit at which you had something ignored but not at new one - that information probably can be used. E.g. sort of in-flight ignores that visible in jj st to inform about either extending explicit ignores or confirming addition of new paths to current change.

P.S. Can think of how git checkout reacts when your untracked file is about to be overwritten by checked out files. In case of jj all files are tracked by default and thus even absence of the file is tracked.

[necauqua]

Hm, so auto-tracking when you are doing changes in a working commit is the main killer feature.

But for my issue with it, how about this: What if jj considered .gitignore not just from the current commit, but from the entire history - or actually just descendant of the current commit? And if you needed to actually add some file in the past you'd have to explicitly track it (instead of explicitly untracking it in the common case you don't need that).

I know this is kind of magical, but the more I think about it the more it makes sense, idk - and the autorebase is magical on it's own, this somehow is kind of even consistent in my head

[martinvonz]

It would be expensive to find the gitignores from all commits, but we could probably index that information. I'm more concerned that it would be unexpected behavior. For example, if you check out a sibling commit where target/ is not ignored, then it would still not be ignored if we only consider descendants.

Maybe it's better to check if the gitignores changed between the old and the new commit and if any untracked files according to the new patterns match the old patterns. If that happen, we could just print a warning about it. We could additionally add the ignores to a per-workspace set of ignores (which we don't support yet). (EDIT: I think this is what @ony suggested.)

[necauqua]

If that happen, we could just print a warning about it

The warning being "those differences are implicitly untracked for this WC, in case your direnv caused GBs of files to generate in target and/or .direnv - add them to gitignore here or explicitly track them, moving to another commit without them ignored (e.g. jj new) will cause them to be tracked in that commit"

^ this is a loose idea, could be refined, for example jj st will have that information ofc

[ilyagr]

Following @ony's suggestion, perhaps when the working copy moves to a new commit, we could track "newly unignored" files by comparing the old .gitignore and the new .gitignore. For example, we could store an extra tree of "newly unignored" files.

Then, the UI could provide ways for dealing with these files. E.g. there could be a command like jj ignored --previously that lists them and jj ignored --previously --restore that gets rid of them. We'd also have to decide whether a modification to a "previously ignored" file makes it no longer "previously ignored".

[kevincliao]

I ran into this today when trying to checkout to a different branch that doesn't contain node_modules. As people have mentioned above at that point it's not possible to run any jj commands. I ended up creating a temporary .gitignore file before being able to jj op restore to a previous checkpoint. Is there a better way to recover when running into this? I wonder if it's possible to have jj op commands still work in this scenario.

[kevincliao]

Ahh ignore my comment, I think I got confused - once there is an option to not auto-add untrack files jj op commands will work again.

[yuja]

Is there a better way to recover when running into this? I wonder if it's possible to have jj op commands still work in this scenario.

You can pass --ignore-working-copy to these commands, but we don't have the last bit to reset the working copy without snapshotting yet.

jj op log --ignore-working-copy  # "jj op log" also works with the current main branch
jj op restore --ignore-working-copy @-
jj workspace update-stale --some-option-to-not-snapshot-before-resetting

[HadrienG2]

Overall, it seems there is no good automated way to handle untracked files when creating commits. One one hand, not tracking them leads to incomplete commits. On the other hand, auto-tracking them leads to commiting of unwanted files. So how would you feel about some variation of the following semi-automated design?

1. By default, interactively prompt before auto-adding files, something like This command will add <list of files> to the current commit, proceed? [y/N].
   • Saying yes follows the current behavior.
   • Saying no aborts the command with an error return value and lets you use jj track and .gitignore as appropriate.
2. Have a way to whitelist sets of files (e.g. source files) so that they are auto-added without a prompt, and not mentioned in prompts when they do occur.
   • This could take the form of a .jjadd file that uses the same glob syntax as .gitignore.
   • The aforementioned prompt would mention the possibility of configuring jj for auto-adding and ignoring.

I think this might strike a good balance between the following concerns:

• Files which we want to track (like source files) eventually get auto-added silently as desired, avoiding incomplete commits and replicating the good parts of the current jj auto-add UX.
• Files which we do not want to track (like object files, target/ directories...) eventually get ignored silently as desired, without undesirable creation of commits that will keep them in the history forever.
• After a short initial configuration period, seeing the prompt becomes an exceptional event and thus leads the user to pause and think, as desired in this situation.

For scripted operation, there should be a way to provide a default answer to the prompt via CLI arguments.

[martinvonz]

I'm personally quite happy with the current behavior (except for the behavior when updating to a commit with different .gitignore). It can be a bit annoying in the beginning, but once you've added the appropriate paths, I find that it works pretty well. Maybe others feel differently. But even if they don't, we may want to make it less annoying for new users by doing something like you suggest.

[ilyagr]

I'm not very happy with the idea of the interactive prompt.

I think that if you edit a .gitignore, any subsequent jj command could trigger this prompt, including jj log. I tend to run an analogue of watch jj log in a tmux pane permanently, and I think this would work very badly with the interactive prompt. Firstly, I'll need to adjust the command to use the "scripted mode". In the "scripted" mode, if the default answer to the prompt is "yes", this goes back to users experiencing auto-add of untracked files. If it's "no", jj's view of the workspace could be out of sync with reality for a while (but, if we go with a prompt, I think this is the better option).

Other UIs will also do an analogue of jj log regularly. Every jj UI (e.g. VS Code plugin) would probably need to have a way of giving this prompt to the user, if we made this interactive.

[HadrienG2]

Ah, yes, there's that. I knew that this design decision of having status commands modify the repository was fishy and going to cause problems someday...

[lf-]

This feature has sadly made me bounce off of jj immediately every time I try it, which is really unfortunate because I keep hearing such good things about it, and want to give it a genuine try.

Every single repository I work on regularly has various testing/strace-log/whatever files in its root. I actually don't mind auto tracking in src/, but in the root it just is not compatible with my workflow.

fwiw a workaround for this that I've not yet checked works on jj might be some kind of terrible thing like so in the .git/info/exclude:

/*
!src/
!Cargo.*

This workaround is quite bad indeed, and I would rather not have to reimplement the git index in .git/info/exclude to be able to use jj, though admittedly it would be with wildcards at least.

[ilyagr]

I have no idea whether this would be helpful you, but here's something that helped me a lot. I can't remember who had suggested it originally; it might be in the FAQ.

• I added _ignore/* to ~/.config/git/ignore (~/.gitignore should also work).

  This is possibly not absolutely optimal (I have been wondering whether /_ignore/ would be better), but works well enough. I actually use _ilyaignore to make the name more unique.

• Create an _ignore subdir in my repo

• Save all weird logs and traces to it

[lf-]

Yup it is in the FAQ or something; I've seen it given as advice before. I just don't like it and it doesn't vibe with how I work, since it would be a whole bunch of extra typing. I could have it be i/ or something, I guess, to reduce typing, but I would still have to remember to do it every time, which feels kind of bad?

[ilyagr]

Inspired by more feedback (https://github.com/martinvonz/jj/discussions/3528#discussioncomment-9148691), perhaps @dpc 's suggestion from that post might work. Perhaps we could have a notion of "untracked" files, like Git, and default files to "untracked", while also auto-updating all the tracked files on each command?

jj status would certainly show any untracked files. jj log could too. It's not quite in the spirit of "everything is a commit" (untracked files would show up as a fake commit in some places), but might work.

One question is what jj diff would do. My first instinct would be to have it act on tracked files only, but complain loudly when there are untracked files. Perhaps each command would do that, I'm unsure.

This would be a huge change, so I almost certainly missed some important considerations.

[dpc]

This would be a huge change

Speaking out of ignorance, I'm guessing jj already needs to compare all worktree files against .gitignore. Right after that it could just compare them against files already tracked in the current change and ignore ones that are not. Plus a command to track a file. And that's kind of it, no? Showing untracked files, etc. seems like a nice-to-have. Deleting a tracked file could work as a "untrack", just like it already does.

Hmm... I guess mv <sometrackedfile> <newlocation> now requires explicit calling "track" on a new location, which is a bit breaking the "immersion", but I think it's fine. And again - this behavior would be optional (but I would suggest making it the default for the sake of newcomers). People that figured out everything could just opt-in into current seamless behavior, which I find elegant and I'm sure I would eventually settle into it just fine, after making sure given repo doesn't produce untracked trash, adding some ./tmp/ to .gitignore and remembering to create my debugging stuff inside it.

[martinvonz]

jj status would certainly show any untracked files. jj log could too. It's not quite in the spirit of "everything is a commit" (untracked files would show up as a fake commit in some places), but might work.

One question is what jj diff would do. My first instinct would be to have it act on tracked files only, but complain loudly when there are untracked files. Perhaps each command would do that, I'm unsure.

If we add support for untracked files, I think it should be pretty much only jj status that shows them. They would just be invisible to every other command. Would that work for the untracked-files proponents?

I guess mv <sometrackedfile> <newlocation> now requires explicit calling "track" on a new location

I don't think so. Almost all commands, and probably also the future jj mv work on commits and just update the working copy to match afterwards.

[dpc]

Would that work for the untracked-files proponents?

:+1: The whole point of not tracking them is so that they don't influence anything.

Almost all commands, and probably also the future jj mv work on commits and just update the working copy to match afterwards.

Correct me if I'm wrong, but currently mv s d would automatically have changes to s and d reflected, possibly even as a rename operation of some kind. I wasn't even aware of jj move which actually seems to exist. Yes jj move would automatically track the destination path in both "explicit" and "seamless" (current) mode. Only "raw" mv while in "explicit" mode would appear to jj as source file being deleted (and thus untracked), and destination file showing up as a new (untracked) file. Which I think is perfectly fine in practice.

[martinvonz]

Correct me if I'm wrong, but currently mv s d would automatically have changes to s and d reflected, possibly even as a rename operation of some kind. I wasn't even aware of jj move which actually seems to exist.

Ah, I just thought you meant a hypothetical jj mv when you said mv. Never mind then :)

[durin42]

If we add support for untracked files, I think it should be pretty much only jj status that shows them. They would just be invisible to every other command. Would that work for the untracked-files proponents?

This actually wouldn't really help me, because my most common use case is for a config file that I want to have sit in the working copy (or a bisection script for git bisect run or similar, I guess) and not have it disappear when the working copy moves around (but also never be added to history.) And typically I don't control the .gitignore because it's someone else's project. :)

[lf-]

I think your use case would work in the git-like untracked files proposal? they're just left alone by every command as if ignored, which would match git behaviour where this does work.

(but if you do need to actually ignore them, .git/info/exclude)

[necauqua]

Inspired by more feedback [..]

I'm fairly certain I talked about having tracked files like this at some point somewhere, eh But yep I'm all for it, I'm still carefully looking at things like vscode git index thing (or actually my old gst -> git status alias) before running jj commands, or doing --ignore-working-copy ¯\_(ツ)_/¯ Perhaps a track-all config?

jj move which actually seems to exist

fyi this one moves files between commits, not like mv :)

[martinvonz]

I don't think it will be a priority for me to work on this any time soon, but I won't object if anyone else sends a PR for this. I haven't thought of any problems with it.

[maan2003]

maybe it should depend on file extension, I wouldn't mind auto tracking *.rs or *.toml or *.sh files