occasional jj hangs

[fowles]

Sometimes all jj commands hang indefinitely. I sadly don't have a great reproduction for this. I usually fix it by typing killall jj and that seems to fix it.

As a bit of background context (no pun intended), I have been seeing intermittent hangs for jj for many versions. I was using watchman's background snapshotting which exacerbated the issue quite a bit. So I turned off background snapshotting when I upgraded to 0.34.0. I also upgraded my terminal to have little status information about jj in the prompt. My prompt is nice and clever so that works happens in a background thread.

❯ cat ~/.config/fish/functions/_tide_item_jj.fish
function _tide_item_jj
    if not command -sq jj; or not jj root --quiet &>/dev/null
        return 1
    end


    set change_id (jj log --no-graph --color=never \
        -r '@' -T 'change_id.shortest(8)')
    string match -qr '(?<changes>#*)(?<bookmark>\w+)' \
      (jj log --no-graph --color=never \
          -r 'closest_bookmark(@)::@' -T '"#" ++ self.bookmarks()')
    set ahead (math (string length $changes) - 1)

    set jj_status (jj log -r@ -n1 --no-graph --color=never -T '
    separate(" ",
        bookmarks.map(|x| truncate_end(10, x.name(), "…")).join(" "),
        tags.map(|x| truncate_end(10, x.name(), "…")).join(" "),
        diff.stat().total_added() ++ "+",
        diff.stat().total_removed() ++ "-",
        if(conflict, "conflict"),
        if(divergent, "divergent"),
        if(hidden, "hidden"),
    )' | string trim)
    _tide_print_item jj "$change_id $ahead⇡$bookmark $jj_status"
end

I don't recall seeing any hangs with this setup in 0.35.0. I definitely see hangs often enough to open this bug in 0.36.0.

Specifications

• Platform: Mac Sequoia 15.6.1
• Version: 0.36.0

[martinvonz]

Please try running lslocks next time it happens. Hopefully that can tell us which locks are being held and by which processes.

[martinvonz]

FYI, @matts1

[fowles]

I thought I hit this again, but in the time it took me to find the mac version os lslocks the command (which was jj diff) eventually finished. So it might not be a hard lock, just a very long delay

[fowles]

Note to self, lslocks doesn't exist on Mac, but these commands might help:

lsof +c 15 -n | grep -i "lock"
sudo fs_usage | grep lock

[KingMob]

FWIW, I'm also seeing this on jj 0.36.0, macOS 26.1.

A jj squash command kept hanging. I tried the lsof and fs_usage commands from above, but found no smoking guns. killall jj had no effect, because there were no jj processes running by then.

Finally, my last squash attempt just worked.

[fowles]

❯ lsof +c 15 -n | grep -i "lock" | grep jj
jj              45769  mfk    5w      REG               1,15          0            85342503 /Users/mfk/dev/pratdiff/.jj/repo/git_import_export.lock
jj              45769  mfk    6w      REG               1,15          0            85342504 /Users/mfk/dev/pratdiff/.jj/working_copy/working_copy.lock
jj              45820  mfk    5w      REG               1,15          0            85342503 /Users/mfk/dev/pratdiff/.jj/repo/git_import_export.lock
jj              45836  mfk    5w      REG               1,15          0            85342503 /Users/mfk/dev/pratdiff/.jj/repo/git_import_export.lock
jj              45838  mfk    5w      REG               1,15          0            85342503 /Users/mfk/dev/pratdiff/.jj/repo/git_import_export.lock
jj              45877  mfk    5w      REG               1,15          0            85342503 /Users/mfk/dev/pratdiff/.jj/repo/git_import_export.lock
jj              45895  mfk    5w      REG               1,15          0            85342503 /Users/mfk/dev/pratdiff/.jj/repo/git_import_export.lock

[fowles]

After 1m 36s, the command completed. At almost precisely the same time that I typed sudo fs_usage | grep lock

[sheremetyev]

Might be lock contention: watchman triggers multiple copies of jj debug snapshot, all of them competing for git_import_export.lock and starving foreground command. @fowles if this is the case, then I think increasing settle parameter in Watchman config, for example to 1 second, would make it less likely.

@KingMob do you have Watchman enabled as well by any chance?

[fowles]

I don't have watchman on at the moment, the background ones are cause by my shell prompt running something in the background to figure out what it wants to display

[matts1]

I have watchman enabled, but watchman.register-snapshot-trigger = false set.

[KingMob]

@KingMob* do you have Watchman enabled as well by any chance?

I don't think so. I experimented with it ages ago, but # core.fsmonitor = "watchman" is currently commented out.

❯ jj config list --include-defaults | grep fs

fsmonitor.backend = "none"
fsmonitor.watchman.register-snapshot-trigger = false

[sheremetyev]

Not sure if that's what is happening, but here is a potential problem: jj process may be stopped while holding git_import_export_lock, which would cause other processes to appear hanging.

When there are divergent operations, jj writes "Concurrent modification detected, resolving automatically." to stderr while holding the lock. If stty tostop is enabled and jj is running in the background, this write triggers SIGTTOU, stopping the process while it still holds the lock.

I'm able to reproduce this way on macOS:

stty tostop
jj new
jj new --at-op=@-
ls .jj/repo/op_heads/heads/
jj status &

After this, running jj log command in another terminal hangs, but running fg here lets jj status complete and unblocks hanging jj log command in another terminal.

@fowles it's an interesting clue that running another command stopped the hang. I think Fish/Tide may have sent SIGCONT to the stopped background jj process (or killed it to refresh the prompt), which released the lock. Could you please check:

• Do you have tostop enabled? stty -a | grep tostop
• Do you have "transient prompt" enabled in Tide?

As a workaround, you could add 2>/dev/null to the jj commands in your _tide_item_jj.fish script to prevent stderr writes from blocking the background process.

[fowles]

I am not particularly familiar with stty, but I think the output indicates that tostop is disable.

❯ stty -a
speed 38400 baud; 59 rows; 190 columns;
lflags: icanon isig iexten echo echoe echok echoke -echonl echoctl
	-echoprt -altwerase -noflsh -tostop -flusho pendin -nokerninfo
	-extproc
iflags: -istrip icrnl -inlcr -igncr -ixon -ixoff ixany imaxbel iutf8
	-ignbrk brkint -inpck -ignpar -parmrk
oflags: opost onlcr -oxtabs -onocr -onlret
cflags: cread cs8 -parenb -parodd hupcl -clocal -cstopb -crtscts -dsrflow
	-dtrflow -mdmbuf
cchars: discard = ^O; dsusp = ^Y; eof = ^D; eol = <undef>;
	eol2 = <undef>; erase = ^?; intr = ^C; kill = ^U; lnext = ^V;
	min = 1; quit = ^\; reprint = ^R; start = ^Q; status = ^T;
	stop = ^S; susp = ^Z; time = 0; werase = ^W;

---

I do not have "transient prompt" enabled.

❯ echo $tide_transient_enabled

---

I think I am already suppressing stderr on my tide prompt.

❯ cat ~/.config/fish/functions/_tide_item_jj.fish
function _tide_item_jj
    if not command -sq jj; or not jj root --quiet &>/dev/null
        return 1
    end


    set change_id (jj log --no-graph --color=never \
        -r '@' -T 'change_id.shortest(8)' 2>/dev/null)
    string match -qr '(?<changes>#*)(?<bookmark>\w+)' \
      (jj log --no-graph --color=never \
          -r 'closest_bookmark(@)::@' -T '"#" ++ self.bookmarks()' 2>/dev/null)
    set ahead (math (string length $changes) - 1)

    set jj_status (jj log -r@ -n1 --no-graph --color=never -T '
    separate(" ",
        bookmarks.map(|x| truncate_end(10, x.name(), "…")).join(" "),
        tags.map(|x| truncate_end(10, x.name(), "…")).join(" "),
        diff.stat().total_added() ++ "+",
        diff.stat().total_removed() ++ "-",
        if(conflict, "conflict"),
        if(divergent, "divergent"),
        if(hidden, "hidden"),
    )' 2>/dev/null | string trim)
    _tide_print_item jj "$change_id $ahead⇡$bookmark $jj_status"
end

(although the first line does &> and not 2> so I will try to fix that)

[sheremetyev]

Thanks @fowles! I think redirection of stderr wasn't in the script previously (I'm looking at the original issue description). Have you seen hangs after adding redirection?

[fowles]

I added redirection for everything except the first line a few weeks ago, so I have definitely seen the error since then. I reworked the script this morning to contain

❯ cat ~/.config/fish/functions/_tide_item_jj.fish
function _tide_item_jj
    if not command -sq jj
      return 1
    end
    set root (jj root --quiet 2>/dev/null)
    if test -z "$root"
      return 1
    end


    set change_id (jj log --no-graph --color=never \
        -r '@' -T 'change_id.shortest(8)' 2>/dev/null)
    string match -qr '(?<changes>#*)(?<bookmark>\w+)' \
      (jj log --no-graph --color=never \
          -r 'closest_bookmark(@)::@' -T '"#" ++ self.bookmarks()' 2>/dev/null)
    set ahead (math (string length $changes) - 1)

    set jj_status (jj log -r@ -n1 --no-graph --color=never -T '
    separate(" ",
        bookmarks.map(|x| truncate_end(10, x.name(), "…")).join(" "),
        tags.map(|x| truncate_end(10, x.name(), "…")).join(" "),
        diff.stat().total_added() ++ "+",
        diff.stat().total_removed() ++ "-",
        if(conflict, "conflict"),
        if(divergent, "divergent"),
        if(hidden, "hidden"),
    )' 2>/dev/null | string trim)
    _tide_print_item jj "$change_id $ahead⇡$bookmark $jj_status"
end

but I usually see the unusually long hangs at ~1w so it will be a while before we can rule it out